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Abstract 

Thiii  research  reviewed  and  validated  important 
req  .^rements  for  the  design  of  an  effective  human/comput er 
interface.  The  requirements  were  derived  from  expert  opinion 
found  in  literature  and  then  many  of  the  requirements  were 
validated  by  protocol  analysis. 

Fourteen  participants,  all  ASD  personnel,  tried  working 
scenarios  using  two  modules  of  the  Program  Manager's  Support 
System.  Their  comments  were  collected,  as  was  an  opinion 
scale,  and  the  data  was  analyzed. 

The  participants'  comments  and  opinions  followed  closely 
with  that  of  the  experts  in  the  literature.  This  study  should 
benefit  ASD  in  their  evaluation  of  this  system  and  it  should 

provide  the  system  developers  ideas  for  program  enhancement. 

...  . 
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AN  ANALYSIS  OF  THE  HUMAN/ COMPUTER  INTERFACE  OF  THE  PROGRAM 
MANAGER  SUPPORT  SYSTEM:  CAPPS  AND  GAT 


I .  Introduction 

In  the  last  decade  there  has  been  a  great  awakening  in 
the  development  and  use  of  personal  computers.  In  business, 
government,  schools,  and  homes  across  the  United  States, 
computers  are  now  a  major  tool  used  for  word  processing, 
data  base  applications,  financial  applications,  education, 
and  a  myriad  of  other  uses .  In  the  technological  rush  to 
develop  computers  and  software  to  present  day  standards, 
designers  often  forget  an  important  aspect :  the 
human-computer  <H/C)  interface. 

The  subject  of  this  research,  the  H/C  interface,  has 
recently  become  an  interest  to  software  designers  and 
computer  professionals.  They  have  begun  to  realize  that  this 
interface  has  been  neglected  in  the  past,  causing  user 
frustration  as  well  as  expensive  computer  programs  and 
systems  to  go  idle  or  to  not  operate  at  their  full 
potential  . 

What  is  an  optimal  H/C  interface?  What  factors  influence 
the  H/C  interface?  Is  there  an  interface  that  will  satisfy 
all  users?  Human  behavior  is  very  complex,  and  there  are  no 
simple  answers  to  these  questions.  This  thesis  represents  an 
attempt  to  analyze  the  building  blocks  of  H/C  interfaces  and 
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to  evaluate  the  interface  used  in  a  current  government 
decision  support  system. 

Definitions 

Merriam  Webster's  Dictionary  defines  the  term  interface 
as  "the  place  at  which  two  independent  systems  meet  and  act 
on  or  communicate  with  each  other"  (Webster,  1974:373).  The 
H/C  interface  must  then  deal  with  the  communication  aspects 
that  occur  when  humans  use  computers. 

The  H/C  interface  adds  several  other  dimensions  to  the 
picture.  Speaking  from  the  human  side  of  the  H/C  interface, 
Rubinstein  and  Hersh  describe  "human  interface"  as  "anything 
the  user  sees,  touches,  or  otherwise  senses  and  interacts 
with  in  a  computer  or  any  other  system  (Rubinstein  and 
Hersh,  1985:234).  Edmonds  suggests  "the  interface  is  that 
part  of  the  system  that  represents  the  user's  model  of  it" 
(Edmonds,  1982:231).  Clarke  views  the  H/C  interface  as  a 
"joining  of  perceptual  and  physical  characteristics  of 
humans  with  the  operational,  functional,  and  organizational 
features  of  computers"  (Clarke,  1986:504,508). 

As  one  can  easily  note,  there  is  no  catch  all  definition 
of  the  H/C  interface.  For  certain  it  involves  the  user's 
senses,  perception,  cognitive  thinking  or  models, 
productivity,  and  decision  making  patterns.  For  simplicity, 

I  choose  to  use  the  definition  proposed  by  Rubinstein  and 
Hersh . 
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The  terms  human  factors  and  ergonomics  are  described  by 
Rubinstein,  Hersh,  and  Martin  as  meaning  the  same  thing:  the 
study  of  how  people  and  machines  interact .  This  involves 
analyzing  those  factors  that  affect  the  welfare, 
satisfaction,  and  performance  of  people  working  with 
man-made  systems  and  equipment,  the  technology  for  creating 
designs  that  work  well  in  human  terms,  or  factors  that  go 
into  the  design  of  the  H/C  interface  to  make  systems  more 
effective  and  to  save  redesign  costs  (Rubinstein  and  Hersh, 
1985:5-7;  Martin,  1987:6). 

Simpson  states  the  term  "User  Friendliness"  is  used 
indiscriminately  by  users  to  describe  something  about  a 
program  that  they  like.  He  defines  user  friendliness 
operationally  in  programs  as  "a  program  with  features  that 
acknowledge  human  factors."  In  general  the  program  will  be 
easy  to  use,  tolerant  of  user  errors,  and  relatively  easy  to 
learn  (Simpson,  1985:2)  . 

Research  Project 

Aeronautical  Systems  Division  (ASD) ,  a  part  of  Air  Force 
Systems  Command,  is  looking  for  software  to  use  as  decision 
making  tools  for  their  program  managers.  They  are  currently 
evaluating  the  Program  Manager's  Support  System  (PMSS) ,  a 
system  developed  by  the  Defense  Systems  Management  College, 
for  possible  use  at  ASD.  This  research  will  aid  in  that 
evaluation  by  analyzing  the  H/C  interface  of  two  modules  of 
the  PMSS,  The  two  modules  to  be  analyzed  are  the  Contract 
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Appraisal  System  (CAPPS),  version  2.10  and  the  Government 
Activity  Tasking  (GAT)  module,  version  1.0. 

The  CAPPS  module  was  designed  to  help  program  managers 
and  their  staff  determine,  quantify,  and  track  the  status 
of  defense  contracts.  CAPPS  uses  Contractor  Performance 
Measurement  guidelines  to  provide  this  information  from  a 
built-in  database.  The  module  also  can  be  used  on  a  variety 
of  Air  Force  microcomputers  in  a  variety  of  modes,  depending 
on  the  available  hardware  and  user  preference. 

The  GAT  module  was  designed  to  help  managers  track 
constrained  financial  and  time  resources  in  government 
activities.  The  module  uses  milestones  derived  from  a  user 
supported  database. 

Research  Objectives 

The  purpose  of  this  research  is  to  determine  the 
requirements  for  an  effective  human-computer  interface,  to 
evaluate  the  H/C  interface  of  two  modules  of  the  PMSS,  and 
to  suggest  possible  H/C  interface  improvements  for  the  PMSS. 
These  objectives  can  be  achieved  by  accomplishing  the 
following  sub-objectives: 

(1)  Define  the  requirements  for  an  effective 
human/computer  interface  from  expert  consensus 
found  in  the  current  literature. 

(2)  Evaluate  the  effectiveness  of  the  PMSS  human- 
computer  interface  through  protocol  analysis  and 
interviewing  techniques. 

Sc ope  and  Limitations 

Software .  This  research  is  limited  to  analyzing  the  H/C 
interface  for  two  of  eighteen  current  and  future  modules  of 
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the  PMSS ,  Other  concurrent  research  will  cover  two  more 
modules,  thus  analyzing  four  of  six  available  modules. 

Research  Goals .  The  primary  purpose  of  this  research  is 
to  study  the  H/C  interface.  Therefore,  no  attempts  will  be 
made  to  insure  the  usefulness  of  the  PMSS  software  on  the 
job.  ASD  is  examining  the  technical  aspects  of  the  usefulness 
of  the  PMSS  separately.  There  will  also  be  no  analysis  of 
hardware  other  than  what  is  needed  to  perform  the  H/C 
interface  analysis. 

Thesis  Overview 

Chapter  two  of  this  study  will  include  the  literature 
review.  The  research  methodology  will  be  explained  in 
chapter  three,  followed  by  a  description  of  the  results  of 
the  study  in  chapter  four.  The  final  chapter  includes 
conclusions  and  recommendations  offered  by  the  researcher. 
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II .  Literature  Review 


Introduction 


This  literature  review  covers  the  following  areas  of 
concern:  general  guidelines,  user  classification,  error 
handling,  display  issues,  documentation,  interaction 
styles,  response  time,  and  data  entry. 

General  Issues  of  the  Human/Comput er  Interface  and  Dialog 
Design 

Before  detailing  each  aspect  of  the  H/C  interface  and 
dialog  design,  there  are  some  general  guidelines  that  are 
worthy  of  review.  First,  Shnoiderman  identifies  "Eight  Golden 
Rules  Of  Dialog  Design"  as  follows  (Shneiderman ,  1987:61-62): 

1.  Strive  for  consistency  in  sequences  of  actions.  This 
includes  terminology  used  in  prompts,  commands,  help 
screens,  and  menus.  Exceptions  should  be  easy  to 
understand  and  very  limited  in  number. 

2.  Enable  frequent  users  to  use  shortcuts.  This  can  be 
accomplished  by  using  macro  facilities, 
abbreviations,  hidden  commands,  and  special  keys. 

There  is  a  tendency  to  want  fewer  and  faster 
interactions  as  the  frequency  of  use  increases. 

3.  Offer  informative  feedback.  For  every  user  action, 
there  should  be  feedback  from  the  system.  Frequent 
and  minor  actions  should  result  in  modest  feedback, 
and  less  frequent  and  major  actions  should  result  in 
more  substantial  feedback  from  the  system. 

4.  Design  dialogs  to  yield  closure.  Action  sequences 
should  be  organized  into  groups  with  a  beginning, 
middle,  and  end.  Feedback  then  gives  a  sense  of 
accomplishment  and  relief  and,  therefore,  signals 
the  user  not  to  worry  about  contingency  plans.  This 
should  prepare  the  user  for  the  next  set  of  actions 
to  undertake. 

5.  Offer  simple  error  handling.  The  user  should  never  be 
able  to  make  a  serious  mistake  by  accident.  Error 
correction  should  be  made  as  simple  as  possible. 
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6.  Permit  easy  reversal  of  actions.  This  has  a  tendency 
to  relieve  anxiety  regarding  errors  and  will  encourage 
the  user  to  explore  unfamili'.r  options. 

7.  Support  an  internal  locus  of  control.  When  users 
become  experienced,  they  often  want  to  feel  in  charge 
of  the  system  with  the  computer  responding  to  their 
actions.  Tedious  data  entry  actions,  surprises  from 
the  system,  the  inability  to  get  desired  information 
from  the  system,  and  the  inability  to  produce  a  wanted 
action  all  lead  to  frustration  and  dissatisfaction. 
Users  want  to  be  the  initiators  of  actions  rather  than 
the  responders  to  actions. 

8.  Reduce  the  short  term  memory  load.  Because  of  human 
information  processing  limitations,  displays  should  be 
kept  simple,  multiple  pages  should  be  condensed,  and 
frequent  windowing  should  be  reduced.  Online  help 
should  be  provided  to  help  the  user  with  lists  of 
commands  and  their  abbreviations. 


Second,  Rubinstein  and  Hersh  discuss  the  general  use  of 
human  language  and  conversation  rules  to  design  more 
effective  dialogs  (Rubinstein  and  Hersh,  1985:59-77).  They 
believe  that  when  people  use  language  in  interacting  with 
machines,  they,  even  though  it  is  not  a  conscious  act, 
behave  as  if  the  rules  for  human  conversation,  language,  and 
meaning  apply.  The  system  should  then  be  designed  to  respect 
the  rules  of  human  conversation  and  language  by  not 
appearing  to  ignore  the  user  when  he/she  asks  for  help.  The 
system  should  provide  appropriate  responses  and  appear  to  be 
"friendly"  by: 

1.  not  interrupting  in  inappropriate  situations, 

2.  responding  with  the  appropriate  amount  of 
information , 

3.  using  language  in  a  consistent  manner, 

4.  teaching  by  example, 

5.  and  avoiding  arbitrary  syntax. 
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The  system  can  also  be  designed  to  enhance  meaning  by  using 
terminology  common  to  the  subject  as  well  as  the  user's 
native  language. 

User  Classification 

People  have  varied  preferences  and  tastes  and  this 
applies  to  their  willingness  to  use,  and  how  they  use 
computers.  Simpson  states  that  any  one  system  will  be  used 
by  a  varied  group  that  ranges  widely  in  sophistication 
(Simpson,  1982:110).  Shneiderman  states  that  a  successful 
H/C  interface  design  depends  on  "a  thorough  understanding  of 
the  diverse  community  of  users  and  the  tasks  that  must  be 
accomplished",  (Shneiderman,  1987:9). 

It  is  generally  found  in  the  literature  that  generic 
classifications  such  as  "novice,"  "expert,"  "frequent,"  and 
"infrequent"  are  used  to  describe  users.  Shneiderman, 

Davis,  and  Olson,  although  using  different  terms,  classify 
users  as  to  the  degree  of  semantic  (application)  and 
syntactic  (computer  system)  knowledge  they  possess 
(Shneiderman,  1987:53-55;  Davis  and  Olson,  1985:530-533). 
Those  definitions  are  as  follows: 

1.  Novice  -  Someone  who  has  little  semantic  and 
syntactic  knowledge  when  first  using  the  system. 

2.  Experts  or  Frequent  Users  -  Users  that  have  a  great 
deal  of  both  semantic  and  syntactic  knowledge. 

3.  Knowledgeable  Intermittent  User  -  Someone  who  has  a 
high  level  of  semantic  knowledge  and  little  syntactic 
knowledge.  This  class  was  proposed  by  Shneiderman 
alone . 


8 


Simpson  believes  that  any  attempt  to  classify  users 
results  in  great  oversimplification,  but  points  out  that 
the  attempt  must  be  made  in  order  to  understand  the  user. 

He  also  states  that  users  will  vary  and  will  change  positions 
based  on  the  following  continuum  (Simpson,  1985:22-27) , 

1.  Sophisticated  Users  -  This  type  of  user  can  typically 
program  software.  They  understand  the  many  software 
design  principles  as  well  as  how  all  the  hardware 
works.  These  users  are  not  afraid  of  software  and  can 
change  software  to  fit  their  own  needs .  Patience  in 
this  group  is  usually  short  and  they  will  sacrifice 
user  friendliness  for  the  sake  of  speeding  up  the 
program.  This  classification  of  users  equates  to 
Shneiderman,  Davis,  and  Olson's  "expert." 

2.  Technical  Professionals  -  Users  that  fall  into  this 
classification  know  little  about,  have  little  interest 
in  knowing  about,  and  may  not  even  like  computers. 

They  will  basically  know  enough  to  turn  on  the 
computer  and  get  to  the  opening  screen  in  the  program. 
Users  in  this  group  usually  do  not  read  the 
documentation  provided  with  the  program  and  have  a 
tendency  to  ignore  or  misunderstand  screen  prompts. 

The  main  interest  of  this  group  is  accomplishing  the 
technical  function,  hence  the  purpose  of  the  program. 
The  concept  of  user  friendliness  is  very  important  to 
this  group,  This  classification  equates  to  the 
"knowledgeable  intermittent"  user  described  by 
Shneiderman . 

3.  Naive  Users  -  This  type  of  user  knows  little  about 
computers  as  well  as  little  about  what  the  program  is 
designed  to  do.  These  users  will  make  every  error 
possible,  so  they  should  be  guided  step  by  step 
through  the  program.  This  classification  is  equivalent 
to  the  "novice"  described  by  Davis,  Olson,  and 
Shneiderman . 

4.  Skilled  Clerks  -  Users  in  this  class  spend  a  great 
deal  of  time  with  computers.  They  are  very  skilled  at 
using  application  programs,  but  that  comes  from 
practice  and  they  should  not  be  considered  as 
sophisticated  users.  They  are  very  much  interested  in 
speed.  This  classification  might  be  considered  a 
subclassification  of  the  "expert"  proposed  by  Davik, 
Olson,  and  Shneiderman. 


9 


Error  Handling 

Errors  impact  all  users  regardless  of  experience  or 
application.  The  errors  not  only  have  an  effect  on  the 
application,  but  on  human  motivation  and  productivity.  Barber 
(1979),  as  reported  by  Shneiderman,  found  that  professional 
workers  who  had  jobs  that  depended  heavily  on  decision 
making,  made  errors  using  the  computer  from  7%  to  4696  of  the 
time  (Shneiderman,  1987:63).  Ledgard  and  others  (1980),  again 
reported  by  Shneiderman,  discovered  that  when  using  a  text 
editor,  novices  made  errors  in  1996  of  their  commands  while 
experts  made  errors  10%  of  the  time  (Shneiderman,  1967:63) . 

The  following  quote  from  Norman  sums  up  several  of  the 
authors'  beliefs  about  errors  and  the  H/C  interface. 

"Errors,  however,  can  be  serious,  both  in  the  sense 
that  they  can  lead  to  serious  mishap,  and  in  the 
social  sense  that  they  frustrate  beginning  users. 

There  is  little  need  for  most  of  these  errors.  Most 
are  system  induced,  the  result  of  inappropriate  system 
design.  Zn  part  these  errors  spring  from  insensitivity 
on  the  part  of  the  designer  to  the  needs  and  functions 
of  users  of  all  abilities,  from  novice  to  expert" 
(Norman,  1983:254). 


Error  Causes  and  Classification .  Shneiderman  believes 
that  a  lack  of  knowledge,  incorrect  understanding,  and 
inadvertent  slips  are  what  cause  most  human  errors 
(Shneiderman,  1987:312).  Norman  agrees  and  classifies  human 
errors  into  three  categories  (Norman,  1983:254-257): 

1.  Mode  error  -  This  occurs  when  a  user  starts  an 

action  that  is  appropriate  in  one  mode,  but  not  in 
that  particular  one.  An  example  of  a  mode  error  would 
be  trying  to  type  text  in  the  command  mode  while  using 
a  text  editor. 


10 


2.  Description  error  -  This  happens  because  different 
actions  have  similar  descriptions.  For  example,  an 
upper  case  letter  causes  a  different  action  than  the 
same  lower  case  letter. 

3.  Capture  error  -  This  will  occur  when  a  sequence  the 
user  is  performing  is  similar  to  another  more  often 
used  sequence  and  overlap  occurs.  For  example,  the 
escape  key  is  used  to  move  about  from  one  function 
to  another  and  a  user  accidently  uses  the  escape  key 
to  try  moving  to  the  next  page. 


Akin  and  Rao  alternatively  classify  human  errors  as 
follows  (Akin  and  Rao,  1985:605-608): 

1.  Command  errors  -  Errors  that  are  caused  by  mistyped 
commands.  Those  errors  can  be  transpositional, 
substitution,  omitted  character,  or  extra  character 
problems . 

2.  Grammatical  errors  of  natural  language  -  Problems 
caused  by  poor  grammar  skills. 

3.  Grammatical  errors  of  system  language  -  Errors  caused 
by  the  wrong  sequence  of  command,  improperly  specified 
keywords,  and  improper  abbreviation^. 


Error  Messages .  Given  the  different  types  of  errors,  how 
should  the  system  respond  to  the  user  error?  The  appropriate 
system  response  usually  involves  an  error  message. 

According  to  Simpson,  error  messages  should  tell  the 
user  there  has  been  an  error,  identify  the  error,  and  show 
the  user  how  to  recover  from  the  error  (Simpson,  19585:142). 
Maguire,  Martin,  Menkus ,  Simpson,  Rubinstein,  Hersh,  and 
Shneiderman  all  recommend  the  following  ways  to  help  the 
user  when  it  comes  to  error  messages. 

1.  Be  consistent  in  form,  position,  and  style  with  all 
error  messages  (Menkus,  1983:13;  Shneiderman,  1987:61; 
Simpson,  1982:118). 
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2.  Make  the  error  messages  helpful  and  descriptive  as  to 
what  the  mistake  was,  where  it  was  made,  and  how  to 
recover  (Simpson,  1982:118). 

3.  State  errors  in  terms  of  the  application  and  not  the 
events  within  the  computer  (Simpson,  1985:141-143), 

4.  Error  messages  should  be  polite,  non-threatening  and 
low-key  .  A  message  from  the  system  such  as  "fatal 
flaw"  or  "nonrecoverable  ...etc"  should  not  be  in  the 
design  (Maguire,  1982:247;  Martin,  1986:12;  Rubinstein 
and  Hersh,  1985:137)  . 

5.  Don't  blame  the  user  in  the  message.  Try  not  to  assess 
blame  at  all.  For  example,  the  message  "Too  many 
characters"  blames  the  user  and  would  be  better  stated 
as  "Five  characters  expected,  but  eight  were  found.", 
(Rubinstein  and  Hersh,  1985:137). 

6.  Don’t  personify  the  computer  by  making  the  computer 
too  talkative  and  trying  to  give  it  a  personality 
(Rubinstein  and  Hersh,  1985:140). 

7.  Make  error  messages  appear  different  from  normal 
messages  (Simpson,  1982:118). 

8.  Make  error  messages  brief  and  explicit  (Rubinstein  and 
Hersh,  1985:137). 

9.  Use  an  audio  signal  to  accompany  any  error  message 
(Menkus ,  1983:13). 


Error  Prevention .  Norman  suggests  several  ways  to  help 
prevent  errors  (Norman,  1983:257): 

1.  Provide  feedback  as  much  as  possible.  If  the  computer 
displays  the  program  mode  that  the  user  is  in,  he/she 
is  less  likely  to  use  commands  designed  for  other 
modes . 

2.  Be  careful  of  similarity  of  response  sequences. 
Different  classes  of  actions  should  have  dissimilar 
menu  patterns  and  command  sequences.  This  will  help 
prevent  capture  and  description  errors. 

3.  Make  all  actions  reversible.  There  are  times  when  an 
action  should  not  be  reversible,  such  as  when  a  user 
saves  or  deletes  a  file.  These  occasions  should  be 
clear  to  the  user  and  all  other  actions  should  provide 
the  user  a  way  to  reverse  an  action  in  case  of  an 
error . 
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Menkus  states  there  is  no  way  to  eliminate  all  errors, 
but  suggests  standardization  of  system  response  messages  as 
the  best  way  to  minimize  errors  (Menkus,  1983:13) .  He 
believes  standardization  is  best  accomplished  by: 

1.  making  each  statement  short,  clear,  positive,  and 
encompassing  a  single  idea, 

2.  making  each  statement  clear  and  complete  in  itself, 
without  reference  to  manuals  or  other  files, 

3.  using  words  that  have  a  commonly  known  meaning  and 
avoiding  jargon, 

4.  using  the  same  words  consistently, 

5.  and  minimizing  the  use  of  verbs. 


Shneiderman  describes  the  following  three  techniques 
for  reducing  errors  (Shneiderman,  1987:64-67); 

1.  Correct  Matching  Pairs  -  Programs  often  use  matching 
pairs  of  symbols  (e.g.,  parentheses)  to  note  special 
operations.  When  the  last  symbol  is  left  out,  an  error 
results.  This  can  be  remedied  by  using  the  carriage 
return  as  universal  ending  symbol  or  by  showing  a  need 
for  a  closing  symbol  on  the  screen  until  the  user 

c  omp lies. 

2.  Complete  Sequences  -  When  a  program  requires  the  user 
to  go  through  a  series  of  steps  to  complete  an  action, 
the  user  can  easily  make  mistakes.  This  can  be  avoided 
by  using  a  single  step  to  accomplish  the  whole  series. 

3.  Correct  Commands  -  Often  a  typographical  error  or  a 
wrong  abbreviation  will  generate  an  error  message  from 
command  line  interfaces.  Systems  that  offer  automatic 
command  completion  after  the  first  few  letters  can 
reduce  this  type  of  error.  Using  another  interface 
style  such  as  the  menu  interface  can  also  reduce 
command  errors . 


Simpson  suggests  a  final  way  that  can  help  reduce 
disastrous  errors.  Users  who  want  to  quit  in  the  middle  of 
a  task  may  be  tempted  to  interrupt  the  program  or  reboot  the 
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system,  causing  data  loss.  Data  loss  can  be  avoided  by 
allowing  the  user  an  escape  from  any  spot  in  the  program 
(Simpson,  1985:143). 

Display  Issues 

The  way  information  is  displayed  often  determines  whether 
the  user  will  use  or  understand  that  information.  Two  aspects 
of  information  display  frequently  mentioned  in  the  literature 
are  consistency  and  human  information  processing  limitations. 
Smith  and  Mosier  emphasize  consistency  in  all  forms  of 
information  display,  because  programs  that  change  format 
midway  are  often  detrimental  to  user  needs  (Smith  and  Mosier, 
1984:93)  . 

Smith,  Mosier,  Davis,  and  Olson  stress  consideration  of 
human  information  processing  limitations  when  designing 
screens  for  information  display.  Too  much  information  on 
one  screen  can  cause  information  overload,  which  can 
adversely  affect  decision-making  (Smith  and  Mosier,  1984:93; 
Davis  and  Olson,  1985:534). 

Rubinstein,  Hersh,  Smith,  Mosier,  Davis,  Olson,  and 
Shneiderman  state  that  related  data  should  be  shown  in 
vertical  (columnar)  fashion,  because  vertical  display  has 
been  shown  to  be  easier  to  read  than  horizontal.  Text 
should  be  left  justified  and  integers  right  justified  with 
decimal  points  lined  up  (Rubinstein  and  Hersh,  1985:153-174; 
Smith  and  Mosier,  1984:93;  Davis  and  Olson,  1985:534-535; 
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Shneiderman ,  1987:70).  To  avoid  confusion  and  concentration 
problems,  several  authors  propose  the  following  guidelines 
for  appropriate  information  display. 

1.  Only  present  the  data  that  is  needed  (Davis  and  Olson 
1985:534;  Shneiderman,  1987:70). 

2.  Number  all  pages  (Shneiderman,  1987:70). 

3.  Identify  output  data  (Rubinstein  and  Hersh,  1985:158) 

4.  Number  from  1  and  measure  from  0  (Rubinstein  and 
Hersh,  1985:158). 

5.  Use  graphs  and  charts  when  possible  (Rubinstein  and 
Hersh,  1985:160;  Shneiderman,  1987:70). 

6.  Avoid  cluttering  the  screen  (Rubinstein  and  Hersh, 
1985:161) . 

7.  Standardize  all  abbreviations,  formats,  and 
terminology  (Shneiderman,  1987:69). 

8.  Insure  information  display  is  compatible  with  data 
entry  (Davis  and  Olson,  1985:534;  Shneiderman, 
1987:70) . 

9.  Separate  sub-items  with  symbols  (ie  -  6/21/60)  (Davis 
and  Olson,  1985:535). 

10.  Left  justify  alphanumeric  data  and  right  justify 
integers  with  decimals  lined  up  (Shneiderman, 

1987:69) . 

11.  The  user  should  be  able  to  control  the  data  display 
(Shneiderman,  1987:70). 

12.  Label  all  information  with  meaningful  labels.  Do  not 
break  up  a  label  unless  necessary  (Davis  and  Olson, 
1985:535)  . 


Attention  Getting .  There  are  times  in  an  interactive 
setting  when  the  user  needs  to  be  notified  of  events, 
including  possible  errors,  feedback  as  to  the  completion  of 
an  action,  impending  problems,  etc.  The  way  to  notify  users 
of  these  events  is  through  employing  attention  getting 
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techniques.  Rubinstein,  Hersh,  and  Shneiderman  suggest  using 
the  subsequent  methods  (Rubinstein  and  Hersh,  1985:163; 
Shneiderman,  1987:71): 

1.  Change  the  display  intensity  for  important  text  or 
numbers . 

2.  Use  boxes  or  borders  to  surround  important 
information . 

3.  Change  font  size  to  emphasize  information. 

4.  Use  color  or  shading  to  make  text  or  numerals  stand 
out  from  other  displayed  information. 

5.  Use  reverse  video. 

6.  Use  auditory  signals. 

It  is  important  to  note  that  overuse  of  the  above  methods 
of  attention  getting  can  cause  crowded  screens  and  or 
confusion  (Shneiderman,  1987:71). 

Using  Color .  The  use  of  color  in  computer  applications 
has  proliferated  greatly  in  recent  times.  It  is  important 
that  color  is  used  correctly  to  enhance  the  H/C  interface. 
Durrett ,  Trezona,  Rubinstein,  Hersh,  Davis,  Olson,  and 
Shneiderman  agree  that  use  of  color  can  be  a  benefit  to  the 
user.  They  also  warn  against  the  misuse  of  color,  which  can 
be  very  distracting  to  users  (Durrett  and  Trezona,  1982:50- 
53;  Rubinstein  and  Hersh,  1985:165-166;  Davis  and  Olson, 
1985:546,548;  Shneiderman,  1987:336-342). 

Davis,  Olson,  and  Shneiderman  state  that  color  is  useful 
for  processes  that  involve  searching,  counting,  or 
identifying  (Davis  and  Olson,  1985:546,548;  Shneiderman, 
1985:336-342).  Shneiderman  further  states  color  adds  accents 
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to  displays  that  are  uninteresting  and  colors  bring  out 
emotional  reactions  in  users  (Shneiderman ,  1987:342). 

According  to  Durrett  and  Trezona,  "Color  vision  is  a 
complex  process  of  three  interacting  variables:  hue, 
brightness,  and  saturation."  Hue  is  the  color,  brightness 
is  the  intensity  of  light  reaching  the  retina  portion  of  the 
eye,  and  saturation  is  the  interaction  of  hue  and  brightness. 
They  state  contrast,  the  brightness  of  the  text  or  numbers 
over  the  background,  and  environmental  lighting  are  important 
in  affecting  screen  contrast  (Durrett  and  Trezona,  1982:50- 
53)  . 


Durrett,  Trezona,  and  Shneiderman  give  the  following 
guidelines  for  using  color: 

1.  Avoid  red/ green,  blue/yellow,  green/blue,  and  red/blue 
pairing  (Durrett  and  Trezona,  1982:51;  Shneiderman, 
1987:341) . 

2.  The  use  of  high  saturation  colors  is  limited  on  less 
expensive  screens,  so  use  the  primary  hues  of  red, 
green,  and  blue  (Durrett  and  Trezona,  1982:51). 

3.  Limit  the  number  of  colors  used  to  four  unless 
experienced  users  prevail,  and  then  use  a  maximum  of 
seven  (Durrett  and  Trezona,  1982:51;  Shneiderman, 
1987:338) . 

4.  Only  use  red  and  green  colors  for  text  and  numerals 
when  the  locations  won't  be  on  the  edge  of  the 
screen  (Durrett  and  Trezona,  1982:51). 

5 .  Use  red  and  blue  when  fast  user  response  is  needed 
(Durrett  and  Trezona,  1982:51). 

6.  Users  often  associate  items  of  like  color  as  having  a 
relationship.  In  screen  displays,  related  items  should 
be  the  same  color  (Shneiderman,  1987:340). 

7.  Color  coding  should  be  a  part  of  the  system  without 
the  user  having  to  make  an  effort  to  get  the  colors 
online  (Shneiderman,  1987:339). 
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8.  The  user  should  be  able  to  turn  off  the  colors 
(Shneiderman ,  1987:339). 

9.  Be  consistent  in  the  use  of  colors  (Shneiderman, 
1987:340)  . 

Graphics .  Information  is  not  limited  to  textual 
situations  only,  but  also  entails  information  of  a 
quantitative  nature.  Many  application  programs  now  use 
graphics  extensively  to  aid  in  decision  making. 

Quantitative  information  can  be  displayed  in  two  forms, 
alphanumerically  or  graphically.  Tullis  states  graphic 
formats  are  superior  to  narrative  formats  because  user 
response  times  were  shorter,  less  training  is  needed  for 
accuracy,  and  overall  user  subjective  ratings  are  higher 
(Tullis,  1981:547).  Simpson  states  graphic  plots  are  very 
effective  for  showing  trends  and  relationships  quicker  than 
alphanumeric  plots  (Simpson,  1985:118). 

Shneiderman  lists  the  following  possible  disadvantages 
of  using  graphics  (Shneiderman,  1987:200). 

1 .  Users  have  to  learn  the  meaning  of  certain  components 
of  graphical  representation. 

2.  Graphics  can  be  misleading  and  cause  faulty 
conclusions  (alphanumeric  display  is  more  precise) . 

3.  Graphics  tend  to  use  a  considerable  amount  of  screen 
space . 

Documentation 

One  of  the  most  important  considerations  in  the  design 
of  the  H/C  interface  is  the  help  provided  to  the  user. 
Nickerson,  Rubinstein,  Hersh,  Houghton,  and  Shneiderman 
define  documentation  to  include  tutorials  (both  online  and 


printed),  online  help,  error  messages,  and  other  types  of 
learning  aids  (Nickerson,  1981:476-477;  Rubinstein  and  Hersh, 
1985:94;  Houghton,  1987:126-128;  Shneiderman,  1987:358-359). 

Clement  suggests  that  the  role  documentation  plays  varies 
and  depends  greatly  upon  the  interface  it  is  describing.  The 
more  natural  the  interface  is,  the  less  extensive  the 
documentation  will  have  to  be,  and  vice  versa  (Clement, 

1984 ;204) . 

Shneiderman  cites  numerous  studies  that  show  printed 
documents  are  superior  to  online.  Printed  text  is  superior 
in  both  user  reading  speed  and  comprehension.  Shneiderman 
states  that  despite  this,  there  is  a  great  attraction  for 
placing  documentation  online  (Shneiderman,  1987:359-362). 

According  to  Shneiderman,  online  documentation  offers 
the  following  advantages  (Shneiderman,  1987:374-376): 

1.  Information  is  available  whenever  the  computer  is 
available . 

2.  Workers  do  not  have  to  allocate  any  work  space  to 
accommodate  bulky  manuals. 

3.  It  is  faster  and  cheaper  to  update  online 
documentation . 

4.  If  electronic  indexing  is  available,  it  takes  less 
time  to  find  information  than  it  does  with  printed 
documentat i on . 

5.  A  computer  screen  can  show  graphics  and  animation  that 
can  explain  complex  actions. 

Shneiderman  also  warns  against  the  misuse  of  online 
documentation  and  proposes  the  subsequent  possible 
disadvantages  (Shneiderman,  1987:375-376): 
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1.  Online  documentation  is  not  as  readable  as  printed 
material  . 

2.  Screens  display  less  information  than  a  sheet  of  paper 
and  paging  through  online  documentation  can  be  slow, 

3.  It  can  be  a  burden  on  short  term  memory  when  one  has 
to  switch  back  and  forth  from  work  to  online 
documentation . 

Interaction  Styles 

To  accomplish  any  work  using  computers,  there  must  be  an 
interaction  between  the  user  and  the  computer.  There  are 
several  ways  users  can  interact  with  computers  to  accomplish 
a  purpose.  All  interaction  styles  are  built  upon  the 
necessity  for  the  user  to  give  commands  for  the  computer  to 
accomplish  a  task.  The  following  interaction  styles  are  the 
three  most  common. 

Command  Interface .  It  is  generally  agreed  that  the 
command  interface  is  the  most  common  interface  style.  The 
command  interface  found  involves  the  user  typing  instructions 
to  the  computer  and  the  computer  carrying  out  those 
instructions . 

Rubinstein  and  Hersh  claim  the  command  interface  is  for 
the  "sophisticated"  user  as  opposed  to  the  novice  (Rubinstein 
and  Hersh,  1985:113).  Shneiderman  describes  the  following 
advantages  of  the  command  interface  (Shneiderman,  1987:136- 
172)  . 

1.  Long  commands  can  be  abbreviated. 

2.  One  command  can  initiate  several  operations. 

3.  Once  learned,  the  commands  are  precise,  thus  making 
this  interface  quicker  than  the  others. 
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4.  Commands  have  an  immediate  impact  on  devices. 

5.  The  user  is  an  initiator  rather  than  a  responder. 

6.  Several  options  are  usually  associated  with  each 
command . 

It  is  generally  thought  that  people  who  are  not  computer 
specialists  have  difficulty  using  computers  because  of  the 
command  interface.  Nickerson  and  Maguire  state  commands  have 
a  formal  syntax  with  inflexible  limits.  Unsophisticated  users 
will  expect  synonyms  to  be  acceptable  and  usually  they  are 
not  (Maguire,  1982:245;  Nickerson,  1981:477).  Shneiderman 
proposes  the  subsequent  disadvantages  to  the  command 
interface  style  (Shneiderman,  1987:136-172). 

1.  The  user  can  confuse  the  syntax  of  the  command 
language  with  that  of  English, 

2.  Infrequent  users  have  a  hard  time  remembering 
commands.  Frequent  users  rarely  memorize  more  than  a 
quarter  of  the  commands , 

3.  Typing  errors  are  easily  made  and  the  need  to  be 
letter  perfect  can  be  frustrating. 

4 .  A  wide  choice  of  inputs  can  be  confusing  to  the 
novice . 

Menu  Interface .  The  second  most  popular  interface  style 
in  use  is  the  menu  style.  According  to  Rubinstein,  Hersh,  and 
Shneiderman,  the  menu  interface  is  excellent  for  users  that 
are  just  beginning  or  have  little  knowledge  of  the  system 
(Rubinstein  and  Hersh,  1985:113;  Shneiderman,  1987:86).  Users 
simply  use  the  arrow  keys  to  reach  a  selection  and  the  return 
key  is  used  to  initiate  the  action. 
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Hollands  and  Merikle  classify  menu  systems  into  three 


categories  (Hollands  and  Merikle,  1987:577-579); 

1.  randomly  organized  topics  for  the  menu  options, 

2.  alphabetized  topics  for  menu  options, 

3.  and  categorically  organized  menu  options. 

They  along  with  Brown,  Rubinstein,  Hersh,  and  Shneiderman 
suggest  balancing  the  category  of  menuing  against  who  the 
users  of  the  program  will  be.  They  also  warn  against  not 
considering  the  user  or  situation  when  using  these  classes 
of  menus  (Hollands  and  Merikle,  1987:579;  Brown,  1982:412- 
418;  Rubinstein  and  Hersh,  1985:119-121;  Shneiderman, 

1987:86) . 

Research  by  Liebelt  and  others,  (1982)  and  McDonald  and 
others,  (1983),  as  reported  by  Hollands  and  Merikle  has  shown 
that  some  users  perform  better  on  certain  classes  of  menus 
(Hollands  and  Merikle,  1987:577-579).  It  was  found  that 
categorical  menuing  was  the  fastest  method,  and  least  likely 
to  cause  problems  for  the  frequent  user.  Alphabetical  menuing 
was  found  to  be  best  for  the  novice  user.  Random  menuing  was 
found  to  be  the  least  desirable  menuing  method.  The  above 
authors  suggest  the  following  guidelines  for  using  menus 
(Hollands  and  Merikle,  1987:577-579); 

1 .  Keep  menus  short . 

2.  Label  all  menus  for  easy  identification. 

3.  Show  the  user  how  he/she  arrives  at  a  particular  menu 
by  displaying  constant  position  information. 


4.  Consider  the  level  of  user  expertise  when  deciding 
which  type  of  menuing  category  to  use. 

5.  Be  consistent  in  the  use  of  titles,  instructions, 
error  messages,  and  status  reports. 

6.  Offer  help  facilities. 

7.  Be  careful  not  to  use  a  series  of  complex  menus  that 
will  slow  down  system  response  time. 

Direct  Manipulation .  This  style  often  involves  the  use 
of  a  pointing  device  such  as  a  mouse,  but  it  is  also 
prevalent  in  most  spreadsheet  programs. 

A  direct  manipulation  interface  uses  objects  on  a  screen 
and  allows  the  user  to  directly  manipulate  them  by  pointing 
at  them  or  typing  directly  into  them.  The  objects  represent 
documents,  files,  or  actions  applicable  to  documents  and 
files.  According  to  Rubinstein  and  Hersh,  direct  manipulation 
is  often  used  in  conjunction  with  the  other  interface  styles 
(Rubinstein  and  Hersh,  1985:123). 

Rubinstein,  Hersh,  and  Shneiderman  state  the  subsequent 
advantages  of  the  direct  manipulation  interface  style: 

1.  Pointing  is  easier  than  saying  (Rubinstein  and  Hersh, 
1985:122)  . 

2.  It  helps  create  literal  conceptual  models  (Rubinstein 
and  Hersh,  1985:121-122). 

3.  It  helps  novices  learn  system  functions  quickly 
(Shneiderman,  1987:201). 

4.  Intermittent  users  can  retain  operational  knowledge 
(Shneiderman,  1987:202). 

5.  Error  messages  are  needed  less  often  (Shneiderman, 
1987:202) . 
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6.  Users  tend  to  experience  less  anxiety  because  the 
system  is  more  easily  understood  and  actions  are  more 
readily  reversed  (Shneiderman ,  1987:202). 

7.  This  interface  can  instill  the  user  with  a  sense  of 
control  over  the  system  (Shneiderman,  1987:202). 


The  above  authors  also  point  out  the  following  possible 
disadvantages  associated  with  the  direct  manipulation  style. 

1.  This  type  of  interface  makes  substantial  demands  on 
a  system.  More  memory,  more  frequent  screen  updates, 
and  more  central  processing  requests  are  a  few  of 
these  demands  (Rubinstein  and  Hersh,  1985:122-123). 

2.  Users  must  learn  the  "icons",  or  graphical 
representations  (Shneiderman,  1987:200). 

3.  This  type  of  interface  can  be  slower  for  more 
experienced  users,  thus  causing  frustration 
(Shneiderman,  1987:200). 

4.  Graphical  representations  take  up  extensive  screen 
space  (Shneiderman,  1987:200). 


Most  of  the  software  available  today  is  a  mixture  of  at 
least  two  interface  styles.  Often  the  different  interface 
styles  complement  each  other  and  allow  for  more  user 
diversity  in  program  applications. 


Response  Time 

Another  important  aspect  of  the  H/C  interface  that 
confronts  users  each  time  they  work  with  computers  is  the 
system  response  time.  Martin  (1973),  as  reported  by  Maguire, 
defines  response  time  as  "the  interval  between  the  operators 
pressing  the  last  key  in  the  input  operation,  and  the 
terminal's  displaying  or  typing  the  first  character  of  the 
response"  (Maguire,  1982:246).  According  to  Davis  and  Olson, 
the  response  time  of  any  particular  system  is  affected  by 
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system  capacity,  the  number  of  users,  and  the  complexity  of 
user  operations  (Davis  and  Olson,  1985:540) . 

In  a  survey  of  over  four  thousand  computer  users,  the 
majority  of  whom  used  microcomputers,  Rushinek  and  Rushinek 
found  response  time  as  the  highest  inducer  of  user 
satisfaction  (Rushinek  and  Rushinek,  1986:597).  Kuhmann  and 
others,  showed  that  different  response  times  affected  pulse 
and  blood  pressure  readings  of  users.  They  also  showed  that 
error  rates  went  up  as  the  response  time  went  down  (Kuhmann 
and  others,  1982:933-940). 

Rubinstein  and  Hersh  state  "Response  is  good,  adequate, 
or  bad  only  in  terms  of  the  users*  perception"  (Rubinstein 
and  Hersh,  1985:149).  Most  authors  generally  agree  that  human 
perception  is  the  key  regarding  response  time.  Novices  wonder 
if  they  have  made  a  mistake  if  the  response  takes  awhile. 
Frequent  users  tire  and  become  upset  if  they  have  to  wait  for 
responses.  Davis  and  Olson  state  response  time  should  not  be 
as  fast  as  possible,  but  should  be  consistent  for  a  given 
level  of  user  expertise  (Davis  and  Olson,  1985:540). 

Miller  (1977),  as  reported  by  Maguire,  suggests 
eliminating  variability,  even  if  it  increases  the  average 
response  time.  He  conducted  a  study  that  showed  users  were 
less  satisfied  and  performed  poorly  on  systems  that  had  high 
variability  in  response  times  (Maguire,  1982:246).  Davis, 
Olson,  and  Shneiderman,  give  the  following  guidelines 
regarding  system  response  times  (Davis  and  Olson,  1985:540; 
Shneiderman,  1987:278); 
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1.  When  there  is  no  penalty  for  making  errors,  users 
will  prefer  to  work  quickly. 

2.  If  the  task  is  easily  comprehended,  users  prefer 
rapid  action. 

3.  When  users  have  had  rapid  performance  in  the  past, 
they  will  expect  it  in  the  future. 

4.  Long  response  times  should  be  announced  by  an  interim 
message . 


Data  Entry 

Data  entry  constitutes  one  of  the  largest  parts  of 
computer  use  and  therefore  can  be  a  large  part  of  the  H/C 
interface.  It  must  be  done  before  the  computer  can  be  used 
as  a  decision  making  tool.  Simpson,  Rubinstein,  Hersh,  Davis, 
Olson,  and  Shneiderman  propose  some  guidelines  regarding  data 
entry  or  input . 

1 .  Use  a  prompt  or  cue  and  make  the  cursor 
automatically  move  to  the  next  location  for  input 
(Davis  and  Olson,  1983:534;  Rubinstein  and  Hersh. 
1985:175;  Simpson,  1982:116). 

2.  The  length  of  the  data  string  entered  should  be  kept 
at  a  minimum  as  to  not  tax  the  human  short  term  memory 
limitation  (Shneiderman,  1987:73). 

3.  Use  a  screen  design  patterned  after  the  paper  form 
that  the  clerk  gets  the  data  from  (Davis  and  Olson, 
1985:534;  Simpson,  1982:116). 

4.  The  user  should  have  the  flexibility  to  enter  data 
however  they  wish  (Rubinstein  and  Hersh,  1985:174; 
Shneiderman,  1987:73), 

5.  Remove  any  redundant  data  entry  (Simpson,  1985:128). 

6.  Allow  abbreviations  in  input  (Rubinstein  and  Hersh, 
1985:174) , 


Simpson  further  suggests  the  subsequent  ways  of  making 
data  input  easier  (Simpson,  1987:127-134;  Simpson,  1982:116- 


1.  Set  limits  as  to  the  length  of  a  data  entry  using  a 
line,  a  pair  of  brackets,  or  some  other  visual  cue. 

2.  Defaults  should  be  displayed  until  changed. 

3.  Feedback  should  be  immediate  so  the  user  can  see  that 
the  data  was  entered  correctly. 

4.  The  operator  should  control  what  data  is  entered  as 
well  as  when  it  is  entered. 

5.  Data  entry  should  be  self  paced. 

6.  Prompting  should  be  adjustable  according  to  the  user's 
desires . 

7.  Allow  users  to  change  their  minds. 

8.  Provide  for  instant  as  well  as  later  error  correction. 

9.  One  area  of  the  screen  should  be  used  for  data  entry 
(unless  a  form-fill-in  screen  is  being  used) . 

Summary 

The  concept  of  the  H/C  interface  is  a  very  complex  one. 
There  are  a  myriad  of  considerations  to  be  made  when 
designing  an  interface.  There  are  complex  relationships 
between  the  mentioned  interface  factors,  the  users,  and  the 
user's  ability  to  accomplish  work  with  computers. 

Shneiderman  states  that  "a  well  designed  system  almost 
disappears,  allowing  the  user  to  concentrate  on  their  work 
or  pleasure"  (Shneiderman,  1987:9).  He  further  states  that  a 
well  designed  H/C  interface  generates  "  positive  feelings  of 
success,  competence,  and  clarity"  with  users  (23:9).  This 
task  is  by  no  means  easy  and  one  must  consider  some  complex 
relationships . 

To  show  some  of  the  many  relationships  a  designer  must 
consider,  an  analysis  of  a  possible  project  might  be  in 
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order.  First,  let  us  consider  the  user.  As  previously 
noted,  users  are  a  diverse  group  with  different  needs  and 
desires.  What  kind  of  users  are  going  to  use  the  program?  Is 
there  an  interface  style  or  combination  of  styles  that  will 
please  all  or  most  of  the  users? 

The  answers  to  the  above  questions  can  directly  relate  to 
issues  concerning:  the  type  of  tasks  to  be  performed,  the 
type  of  equipment  needed  to  run  the  program,  the  type  of 
training  needed  for  the  users,  and  the  overall  expense  of  the 
program . 

The  tasks  to  be  performed  determine  needs  in  the  areas  of 
information  display  and  input,  system  response  time,  error 
prevention  and  handling,  documentation,  and  interface  styles. 
For  instance,  users  in  a  nuclear  power  plant  might  need  a 
system  that  provides:  quick  response  times,  display  screens 
that  use  color  and  sounds  for  warnings,  extremely  neat 
screens  that  properly  use  graphics  and  other  information 
characters,  ways  to  prevent  errors,  ways  to  quickly  overcome 
errors  through  online  and  off-line  documentation,  and  precise 
input  techniques.  Other  types  of  tasks  need  other 
considerations . 

If  the  users  already  have  equipment,  there  may  be 
limitations  as  to  interface  styles,  response  times,  and  data 
display  and  input.  The  designer  will  have  to  design  according 
to  the  equipment . 

This  by  no  means  covers  all  the  H/C  interface  issues 
that  a  designer  might  face,  but  hopefully  one  can  see  the 
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complex  relationships  that  are  found  in  H/C  interface  issues, 
and  much  time  and  care  should  be  put  into  the  design  of  a 
program  to  insure  the  best  H/C  interface  possible. 
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III.  Methodology 

Introduction 

This  study  of  the  H/C  interface  of  the  CAPPS  and  GAT 
modules  of  the  PMSS  involved  the  use  of  protocol  analysis 
and  interview  techniques.  According  to  Emory,  the  interview 
technique  has  the  following  inherent  strengths  (Emory, 
1980:294) . 

1.  This  method  allows  data  to  be  collected  at  the  time  of 
occurrence  and  therefore  is  not  forgotten. 

2.  Common  information  is  not  overlooked  or  seen  as 
irrelevant . 

3.  This  method  allows  the  researcher  to  collect  more 
detailed,  better  quality,  and  a  greater  quantity  of 
information . 

Ericsson  and  Simon  state  some  basic  assumptions  in  using 
the  protocol  analysis  form  of  research.  Those  assumptions 
important  to  this  study  are  listed  below  (Ericsson  and  Simon, 
1984:9)  . 

1 .  Verbal  behavior  is  a  type  of  behavior  that  can  be 
observed  and  analyzed  like  any  other  behavior. 

2.  The  thinking  processes  that  are  behind  verbalizations 
are  a  subset  of  the  thinking  processes  that  cause  any 
other  behavior  behavior. 

3.  A  single  invalid  verbal  statement  as  with  other 
behaviors  should  not  cause  the  discarding  of  verbal 
rep'''rts  in  general . 

With  the  above  strengths  of  interviewing  and  the  assumptions 
of  protocol  analysis  in  mind,  this  study  of  the  H/C  interface 
of  the  PMSS  should  be  considered  valid  because:  the 
information  gathered  by  protocol  analysis  should  be  the  same 
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as  that  gathered  by  any  other  method;  protocol  analysis 
allows  more  data  to  be  collected  because  it  lessens  the 
amount  forgotten;  and  the  data  that  is  collected  should  be 
more  detailed  and  of  a  better  quality. 

Limitations 

Emory,  Ericsson,  and  Simon  have  posed  some  warnings 
regarding  weaknesses,  issues,  or  problems  with  interviewing 
and  protocol  analysis .  Emory  poses  the  following  weaknesses 
with  interviewing  (Emory,  1980:286-288). 

1.  The  setting  up  and  actual  data  collection  is  very  time 
consuming . 

2,  If  participants  do  not  easily  speak  their  minds, 
surface  indicators  must  be  used  to  reach  conclusions. 

Ericsson  and  Simon,  state  the  following  issues  are 
worthy  of  thought  before  embarking  into  a  protocol  analysis 
effort  (Ericsson  and  Simon,  1984:2-6). 

1.  Some  researchers  express  doubts  about  verbal  data. 

Some  have  said  it  is  good  for  surveys,  but  it  is 
difficult  to  objectively  measure. 

2.  Data  is  hard  to  extract  from  behavior.  Illegitimate 
introspection  is  hard  to  distinguish  from  data. 

3.  Some  researchers  believe  verbal  reports  are  "soft" 
data  rather  than  "hard".  They  feel  that  verbal  reports 
may  be  more  theory  than  data. 

As  noted  above,  this  type  of  research  is  inherently  time 
consuming.  Because  of  this  and  time  constraints,  there  were 
only  seven  participants  involved  with  each  of  the  modules. 
Because  of  this  small  sample  size,  all  statistics  are  limited 
to  those  of  simple  summary  type. 
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Besides  the  above  listed  assumptions,  another  assumption 
was  also  made  that  the  participants  in  the  study  were 
knowledgeable  intermittent  users  in  that  they  knew  the 
application  subject,  but  knew  nothing  about  how  the  PMSS 
works . 

Participants 

The  participants  in  all  the  modules  were  ASD  personnel. 
Participants  in  military  lower  management  positions  were  all 
officers  between  the  ranks  of  Second  Lieutenant  and  Captain. 
The  lower  level  managers  managed  small  parts  of  large 
programs  and  supervised  few  if  any  personnel.  Military  middle 
level  managers  in  this  study  held  the  rank  of  Major,  and  they 
tended  to  manage  larger  projects  and  to  supervise  more 
people.  Civilian  engineering  personnel  were  between  the 
grades  GS  nine  through  twelve,  were  involved  with  ASD 
projects,  and  typically  did  not  supervise  anyone.  Civilian 
middle  managers  were  between  the  grades  of  GS  twelve  and 
fourteen;  typically  their  duties  were  similar  to  military 
middle  level  managers. 

All  of  the  participants  were  involved  with  providing 
decision  support  for  upper  management.  The  participants  all 
had  experience  with  Air  Force  microcomputers,  including  the 
Zenith  100,  150,  and  248  models.  Their  experience  included 
applications  use  with  spreadsheets,  wordprocessing,  and 
graphics.  These  applications  were  used  mainly  for  decision 
support  for  themselves  and  upper  level  management. 
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There  were  four  military  and  three  civilian  participants 
using  the  GAT  module,  with  the  military  representing  lower 
management  and  civilian  representing  clerical  help.  The 
remaining  civilians  were  considered  engineering  staff.  The 
participants  using  the  CAPPS  module  included  four  military 
middle  managers,  one  military  lower  manager,  one 
civilian  middle  manager,  and  one  civilian  engineer.  This 
participant  mixture  reasonably  represents  the  ASD  users  of 
the  PMSS. 

Problem-Solving  Tasks 

A  scenario,  consisting  of  several  tasks,  was  developed 
for  the  protocol  collection  for  each  module  (see  appendices 
A  and  B) .  The  tasks  were  designed  to  specifically  test  each 
phase  of  the  module  so  the  participants  would  be  exposed  to 
all  aspects  of  the  module  H/C  interface.  The  scenarios  were 
evaluated  by  the  ASD  project  officer  to  insure  the  content 
was  similar  to  that  which  could  be  found  in  everyday  system 
program  office  work. 

GAT  Module .  The  scenario  for  the  GAT  module  told 
participants  they  would  be  changing  data  for  engineering 
change  proposals  (ECP' s)  fcr  a  program  currently  in  progress. 
The  information  could  then  be  used  for  progress  checking  or 
for  making  reports.  The  tasks  were  designed  as  shown  below. 

1.  Task  1  -  This  task  involved  editing  an  existing 
database.  It  was  designed  to  test  editing  features 
(data  input  and  error  correction  issues) ,  ease  of 
movement,  and  information  display  issues. 

2.  Task  2  -  This  task  was  designed  to  test  ease  of 
movement  and  information  display. 
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3.  Task  3  -  This  task  was  designed  in  the  same  manner  as 
task  1 . 

CAPPS  Module .  The  scenario  for  the  CAPPS  module  put 
participants  in  the  role  of  a  program  office  staff  person 
working  on  a  fictional  new  program.  Their  job  involved  two 
things .  The  first  was  tracking  the  progress  of  the  program  by 
changing  input  data  and  the  second  involved  making  reports 
for  upper  management.  The  tasks  were  designed  as  shown  below. 

1 .  Task  1  -  This  task  was  designed  mostly  to  test 
information  display  and  movement  issues. 

2.  Task  2  -  This  task  was  designed  to  test  the  same 
issues  as  task  1 . 

3.  Task  3  -  Issues  concerning  error  correction,  data 
input,  and  ease  of  movement  went  into  the  design  of 
this  task. 

4.  Task  4  -  This  task  was  designed  to  test  data  input  and 
ease  of  movement. 

Participant  Pr eparation 

The  participants  were  briefed  on  a  variety  of  points 
prior  to  the  protocol  collection  and  interview  (see  appendix 
C) .  The  purpose  of  the  briefing  was  to  describe  the  study, 

ASD  interests  regarding  the  study  and  the  I^SS,  to  determine 
participant  scheduling  and  module  preferences,  and  to  answer 
any  participant  questions. 

After  scheduling  the  participants  to  a  particular  module, 
the  documentation  and  scenario  were  given  to  them.  This  was 
done  at  least  a  day  and  one  half  before  the  protocol 
collection.  The  participants  were  instructed  to  preview  the 
scenario  and  the  documentation.  The  preview  was  not  to  be  in 
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time  consuming  detail,  but  was  to  be  done  to  familiarize  the 
participant  with  general  program  capabilities  and  movement 
patterns , 

Eguipment 

The  PUSS  modules  were  run  on  a  Zenith  248  IBM  compatible 
microcomputer.  It  was  equipped  with  a  color  monitor,  enhanced 
graphics  adapter,  640k  random  access  memory,  a  standard  IBM 
keyboard,  a  20  megabyte  hard  drive,  and  one  floppy  disk 
drive . 

To  aid  in  the  data  collection,  a  voice  activated 
microcassette  was  used  during  the  protocol  collection  and 
interview . 

Test  Site  Lav out /Conditions 

ASD  provided  a  partitioned  office,  approximately  12  feet 
by  12  feet,  to  conduct  the  protocol  collection  and  interview. 
It  was  surrounded  by  other  offices,  of  which  half  were 
vacant.  The  office  contained  two  desks,  one  of  which  was  used 
for  the  computer  (see  appendix  D) . 

During  the  protocol  collection  and  interview,  the 
background  noise  was  typical  of  office  type  work  and 
consisted  of  muffled  voices  in  other  offices  and  the  hallway, 
ringing  telephones  in  other  offices,  and  the  constant  noise 
of  the  microcomputer  fan.  The  temperature  was  about  78 
degrees  F.  with  the  air  conditioning  provided.  The  lighting 
was  florescent  and  provided  ample  light  for  reading. 
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Opinion  Scale 

The  opinion  scale  consisted  of  eleven  questions  taken 
from  "Designing  the  User  Interface"  on  various  issues  dealing 
with  the  H/C  interface  (Shneiderman ,  1987:400:404).  Each 
question  provided  participants  a  scale  from  zero  to  six,  with 
zero  being  the  worst  rating  and  six  being  the  best,  to  best 
fit  their  opinions.  For  more  detailed  information  on  the 
opinion  scale,  see  Appendix  G. 


Protocol  Collection  and  Interview 

Upon  arrival,  the  participants  were  greeted  and  made 
comfortable  at  the  desk  with  the  microcomputer.  To  start  the 
procedure,  the  participants  were  given  an  overview  of  what 
was  going  to  happen  during  the  procedure.  They  were  told  the 
session  would  include  an  agreement  procedure,  followed  by  a 
warm-up  game,  the  module  scenario  session,  an  opinion  scale 
session,  and  a  five  question  wrap-up.  The  detailed  procedure 
is  listed  below. 

1.  The  participants  were  given  a  statement  of 
agreement  with  the  researcher's  signature  to 
emphasize  their  anonymity  and  to  encourage  them  to 
voice  their  opinions  (see  appendix  E)  . 

2.  The  participants  were  then  introduced  to  a  computer 
game  as  a  warm-up  to  the  computer  and  the  protocol 
collection  procedure.  The  instructions  for  play  and 
the  interface  issues  to  look  for  were  explained  to 
each  participant.  For  a  detailed  description  of  the 
game  and  the  questions  asked,  see  Appendix  F. 

3.  The  participants  were  then  instructed  to  load  the 
particular  PUSS  module  being  tested  and  to  begin 
working  the  scenario.  They  were  encouraged  to  voice 
opinions  about  any  part  of  the  program  and  to  feel 
free  to  use  the  documentation  provided. 
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4.  If  during  the  protocol  collection  a  participant 
appeared  puzzled  or  commented  in  a  voice  that  the 
recorder  could  not  pick  up,  they  were  asked  to  repeat 
or  tell  what  they  liked  or  disliked  about  that 
particular  part  of  the  program  module.  The  researcher 
was  also  taking  notes  during  the  entire  process. 

5.  At  the  conclusion  of  the  scenario,  the  participants 
were  given  a  subjective  rating  scale  (see  appendix 
G) .  They  were  given  instructions  on  how  to  fill  out 
the  form  and  were  told  to  feel  free  to  make  added 
comments  by  either  writing  them  down  or  voicing  them. 

6.  The  participants  were  then  asked  some  summary 
questions  to  provide  details  as  to  their  experience 
with  computers  and  the  usefulness  of  the  module  in 
relationship  to  their  current  or  past  jobs.  The 
questions  can  be  found  in  Appendix  H. 
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Introduction 


IV .  Analysis  and  Results 


At  the  conclusion  of  the  protocol  analysis  and  data 
gathering  portion  of  the  this  research,  the  data  was 
analyzed.  A  microcomputer  statistics  package,  called 
Interactive  Statistical  Programs,  was  used  to  calculate  the 
summary  statistics  from  the  participant  opinion  scale  (see 
appendices  I  and  J) .  All  oral  participant  comments  were 
transcribed  from  tape  and  filtered  to  remove  comments  that 
did  not  relate  to  the  issues  at  hand.  The  transcribed 
comments  (see  Appendices  K  and  L)  are  organized  by  the  task 
structure  given  in  the  participant  scenarios. 

Participant  comments  on  the  two  PMSS  modules  will  be 
analyzed  separately  using  the  major  areas  of  the  H/C 
interface  listed  in  the  literature  review  as  a  guide.  The 
order  of  topics  in  the  literature  review  will  also  be  the 
order  by  which  the  topics  will  be  covered  in  this  chapter 
except  that  only  those  interface  areas  employed  in  each 
module  and  commented  upon  by  the  participants  will  be 
included.  Those  areas  of  the  H/C  interface  that  do  not  appear 
to  fit  under  one  of  the  specific  classifications  will  be 
discussed  in  the  general  issues  section,  which  includes 
Shneiderman ' s  "Eight  Golden  Rules  of  Dialog  Design."  Comments 
that  do  not  easily  fit  into  one  of  those  design  issues  will 
be  discussed  in  a  miscellaneous  section. 

The  results  (ie  -  frequency  distribution  and  high  to  low 
ratings)  from  each  question  in  the  participant  opinion  scale 
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will  be  illustrated  throughout  the  chapter.  The  mean  on  a 
scale  of  seven  and  standard  deviation  from  each  question  will 
also  be  identified.  It  is  important  to  note  that  questions  in 
the  opinion  scale  regarding  online  help  and  graphics  only 
apply  to  the  CAPPS  module. 

GAT  Results 

General  Issues 

Ease  of  Movement.  Five  of  the  seven  participants 
made  comments  regarding  the  ease  of  movement  in  the  GAT 
module,  The  majority  of  the  comments  dealt  with  the  use  of 
arrow  keys  for  movement  within  the  program.  Some  comments 
concerning  the  use  of  the  space  bar  dealt  more  with  display 
issues  and  will  be  discussed  in  that  section. 

One  participant  suggested  using  the  tab  key  to  shift  from 
field  to  field.  Three  participants  suggested  the  program 
would  be  more  efficient  if  changed  to  allow  the  use  of  all 
four  arrow  keys  throughout  the  program  for  scrolling  up  and 
down  as  well  as  side  to  side.  One  participant  expressed  no 
preference  for  the  use  of  right/left  or  all  arrow  keys,  but 
did  suggest  making  the  program  consistent  one  way  or  the 
other.  This  participant's  opinion  is  in  agreement  with  the 
literature  (Shneiderman ,  1987;61).  The  literature  does  not 
specify  any  technique  as  the  best,  but  it  does  support  the 
need  to  use  one  technique  consistently. 

One  question  in  the  opinion  scale  concerned  the  ease  of 
movement  throughout  the  module.  The  mean  participant  rating 
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in  this  area  was  4.57  with  a  standard  deviation  of  1.9. 
Figure  1  details  opinion  scale  results. 


Participant  Ratings 

Fig.  1.  GAT  Ease  of  Movement  Histogram 

Consistency  in  Sequences  of  Actions .  Disregarding 
the  comment  about  consistency  in  arrow  key  use  already 
discussed,  this  issue  surfaced  in  the  comments  of  another 
participant.  The  participant  commented  that  when  trying  to 
save  a  report  format,  he  tried  to  exit  first,  which  was 
inappropriate  at  the  time.  The  participant  stated  that  he  was 
programmed  to  exit  before  saving  because  that  is  the  way  it 
was  done  earlier  in  the  program  in  a  similar  situation. 
According  to  the  literature,  it  should  be  no  surprise  that  a 
participant  had  a  problem  at  this  point.  The  sequences  of 
actions  were  not  consistent;  therefore,  problems  can  occur 
(Shneiderman ,  1987:61). 

Offer  Informative  Feedback.  The  opinion  scale 
questioned  participants  about  the  clarity  of  the  GAT  module 
system  massages.  The  mean  participant  rating  was  4.28  with  a 
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standard  deviation  of  1.38.  Figure  2  details  the  range  and 
distribution  of  these  ratings. 


Participant  Ratings 

Fig.  2.  GAT  System  Messages  Histogram 


Error  Handling .  Most  of  the  comments  in  this  area  were 
very  closely  related  to  the  design  of  the  screen  display  and 
will  be  discussed  along  with  screen  display  issues.  The  only 
exception  was  a  comment  by  one  participant,  who  stated  you 
had  to  realize  where  you  were  (ie,  which  screen  was  being 
displayed)  to  be  able  to  correct  errors  effectively. 

The  ability  to  correct  errors  was  the  topic  of  another 
question  in  the  opinion  scale.  The  mean  participant  rating 
was  4.28  with  a  standard  deviation  of  1.49.  Figure  3  further 
details  participant  ratings  regarding  error  correction. 


Frequency 
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Display  Issues 

Col  or .  Five  participants  stated  opinions  about  the 
use  of  color  in  the  program.  Three  participants  said  they 
preferred  using  the  default  colors  of  white  or  grey  on  a 
background  of  dark  blue.  Two  of  those  three  participants 
stated  a  preference  for  light  blue  writing  over  the  grey  or 
white  on  the  default  dark  blue  background,  although  the 
program  was  not  bad  with  the  default  white  that  turns  grey 
when  entered.  One  participant  stated  that  the  program  should 
have  provided  more  color  options  because  the  equipment  could 
support  more  of  a  variety.  Once  again,  the  GAT  module  does 
not  offer  something  that  is  available  in  many  application 
programs .  This  makes  the  GAT  module  inconsistent  with  what  is 
already  available  for  use.  It  is  also  important  to  note  that 
one  participant  said  that  he  did  like  the  options  available. 

Another  opinion  scale  question  rated  how  the  colors  used 
affected  contrast.  The  mean  participant  rating  was  5.14  with 
a  standard  deviation  of  .69.  Figure  4  describes  participant 
responses  for  the  quality  of  GAT  color  contrast. 


Participant  Ratings 
Fig.  4.  GAT  Contrast  Histogram 
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The  general  use  of  color  was  the  topic  of  another 
question  in  the  opinion  scale.  The  mean  participant  rating 
for  the  overall  use  of  color  was  5.43  with  a  standard 
deviation  of  .79.  For  the  distribution  of  participant  ratings 
and  the  highest  to  lowest  ratings,  refer  to  Figure  5. 


Participant  Ratings 

Fig,  5.  GAT  Overall  Use  of  Color  Histogram 

Screen  Layout .  Four  participants  made  comments 
regarding  the  layout  of  information  presented.  Three 
participants  did  not  care  for  the  header  listing  order  and 
suggested  the  headers  in  the  report  generator  (see  Figure  6) 
be  listed  in  alphabetical  order  to  avoid  confusion  and  help 
users  find  the  needed  headers  more  quickly.  These  opinions 
are  in  agreement  with  the  literature.  Since  the  header  option 
display  is  essentially  a  menu,  research  has  shown 
alphabetical  organization  to  be  more  effective  than  random 
ordering  (Hollands  and  Merikle,  1987:578). 

Two  participants  expressed  some  concern  over  the  headers 
in  the  report  generator  being  too  large  for  the  display  box. 
Because  the  field  size  of  some  of  the  headers  was  too  large 
to  be  fully  displayed,  the  participants  were  concerned  over 
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the  quality  of  the  printed  report.  According  to  the 
literature,  consistency  should  be  emphasized  when  it  comes  to 
information  display  (Smith  and  Hosier,  1984:93),  It  should 
then  be  no  surprise  that  headers  displayed  one  way  at  the  top 
of  the  screen  and  another  at  the  bottom  could  cause  problems. 
While  editing  numbers,  one  participant  said  the  program 
ought  to  act  like  a  calculator  by  starting  on  the  right  edge 
and  forcing  the  number  out  as  it  goes.  This  participant  is 
also  in  agreement  with  the  literature  (Shneiderman ,  1987:69). 

Two  other  participants  commented  that  they  would  prefer 
leaving  information  in  subject  fields  and  changing  only  that 
which  needed  change,  instead  of  reentering  all  of  the 
information  in  a  field.  These  comments  agree  with  the 
literature  in  two  ways.  First  of  all,  redundant  data  entry 
should  be  avoided  (Simpson,  1985:128),  Although  part  of  the 
data  needed  editing,  most  of  it  was  correct;  therefore, 
typing  it  again  is  redundant.  Secondly,  users  should  have 
flexibility  in  how  they  enter  data  (Rubinstein  and  Hersh, 
1985:174;  Shneiderman,  1987:73),  The  participants  have  no 
options  while  editing  in  the  GAT  module. 

Two  participants  also  said  it  was  a  little  strange  and 
confusing  having  the  previous  numbers  stay  in  fields 
during  the  editing  process,  rather  than  blanking  the  whole 
field  until  replacements  were  entered  by  striking  the  "enter" 
button.  The  blanking  feature  is  available  in  some  application 
programs , 
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One  participant  commented  on  the  overall  screen  display 
other  than  by  a  ranking  on  the  opinion  scale.  The  participant 
stated  the  screen  display  was  neat  and  very  orderly. 

The  opinion  scale  measured  what  the  participants  thought 
of  the  neatness  of  the  screen  layout.  The  mean  participant 
rating  in  this  area  was  5.29  with  a  standard  deviation  of 
.76.  The  distribution  and  rating  range  are  shown  below  in 
Figure  7. 


Frequency 


Fig.  7.  GAT  Screen  Display  Histogram 


Documentation .  Besides  expressing  opinions  about  the 
documentation  in  the  opinion  scale,  three  participants 
contributed  more  comments.  One  participant  said  the 
documentation  was  good,  but  maybe  a  little  too  simple.  He 
further  added  that  in  some  cases  simplicity  might  be  the  best 
approach.  Another  participant  stated  a  liking  for  the  users' 
manual,  especially  the  step  by  step  approach  of  what  to  do, 
why  it  needs  to  be  done,  and  how  to  do  it.  Finally,  one 
participant  said  the  documentation  was  fairly  easy  to  use,  in 
fact  much  easier  than  the  documentation  that  comes  with  Air 
Force  microcomputers.  The  literature  treats  written 
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documentation  as  a  continuation  of  the  H/C  interface  and  as 
such,  it  is  required  to  accommodate  all  levels  of  users.  The 
participant  that  thought  the  documentation  was  too  simple  was 
an  expert  and  he  realized  the  documentation  needed  to  be 
simple  to  aid  the  other  classes  of  users. 

Two  questions  in  the  opinion  scale  dealt  with 
documentation  issues.  One  covered  the  overall  layout  and  how 
easy  it  was  to  use  and  the  other  covered  the  participants 
overall  opinion  of  the  helpfulness  of  the  written 
documentation.  The  mean  participant  rating  of  the  layout  was 
4.72  with  a  standard  deviation  of  1.38.  Figure  8  shows  the 
frequency  distribution  of  participant  ratings. 


Participant  Ratings 


Fig.  8.  GAT  Written  Documentation  Layout  Histogram 

The  mean  participant  rating  concerning  overall  opinions 
about  the  written  documentation  was  5.14  with  a  standard 
deviation  of  .69.  Figure  9  details  the  range  and  distribution 
of  those  ratings. 
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Participant  Ratings 

Fig.  9.  GAT  Overall  Written  Documentation  Histogram 

Response  Time .  Two  participants  made  comments  about  the 
system  response  time.  Both  said  the  response  time  was 
excellent,  especially  since  a  floppy  disk  drive  was  used 
instead  of  a  hard  drive.  The  literature  suggests  response 
time  is  good  or  bad  only  in  terms  of  the  users'  perception 
(Rubinstein  and  Hersh,  1985:149).  Since  the  participants  did 
rate  response  time  as  favorable,  the  response  time  is 
appropriat  e . 

The  opinion  scale  rated  the  participants'  opinions  on  the 
system  time  used  in  response  to  user  commands.  The  mean 
participant  racing  for  response  time  was  5.43  with  a  standard 
deviation  of  .79.  Figure  10  gives  more  summary  statistics. 


Participant  Ratings 
Fig.  10.  GAT  Response  Time  Histogram 
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Data  Entry .  The  comments  concerning  the  editing  process 
discussed  with  display  issues  could  have  also  been  applicable 
to  data  entry  issues.  There  were  no  questions  dealing 
specifically  with  data  entry  on  the  opinion  scale  since  so 
little  data  was  entered  during  the  scenario  exercises. 

CAPPS  Results 

General  Issues 

Ease  of  Movement .  The  issue  of  using  a  space  bar 
instead  of  arrow  keys  for  general  movement  also  arose  in  the 
CAPPS  module.  Four  participants  commented  on  this  issue.  One 
participant  said  he  did  not  like  using  the  space  bar  and  the 
back  space  keys  to  move  within  the  program,  but  would  rather 
use  the  arrow  keys  all  of  the  time.  Three  participants  said 
nothing  was  wrong  with  using  the  space  bar,  it  was  just 
different  from  the  way  other  programs  handle  the  movement 
issue.  Two  of  the  three  participants  suggested  some  mistakes 
they  made  were  caused  by  a  lack  of  consistency  in  using  the 
space  bar  or  the  arrow  keys .  They  suggested  the  program 
should  be  consistent  throughout.  As  with  the  GAT  module,  the 
CAPPS  module  goes  against  the  literature  by  disregarding 
consistency  of  actions. 

Two  participants  suggested  there  should  always  be  an 
option  in  the  CAPPS  menu  system  of  going  back  to  the  previous 
page.  Although  they  were  the  only  participants  to  suggest 
this,  they  were  also  the  only  ones  to  accidently  go  one 
screen  too  far  while  trying  to  accomplish  tasks.  The 
literature  agrees  with  these  comments  and  suggests  actions 
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that  are  easily  reversed  can  lessen  user  anxiety  about  errors 
as  well  as  encourage  users  to  explore  the  unfamiliar 
(Shneiderman ,  1987:62). 

Again,  the  opinion  scale  rated  the  participants'  opinions 
on  how  easy  and  time  efficient  it  was  to  move  throughout  the 
CAPPS  module.  The  mean  participant  rating  was  4.2  with  a 
standard  deviation  of  .97.  Figure  11  describes  the 
distribution  and  range  of  the  ratings. 

Frequency 


Fig.  11.  CAPPS  Ease  of  Movement  Histogram 

Offer  Informative  Feedback.  System  messages  are 
feedback  for  the  user,  and  one  participant  commented  about  a 
problem  in  this  area.  While  trying  to  use  the  "pz'evious 
screen"  option,  the  participant  stated  the  system  message 
telling  him  that  he  could  not  go  back  further  was  too  short 
in  duration.  He  had  to  press  the  button  three  times  to  read 
the  full  message.  The  participant  had  a  reason  to  comment 
according  to  the  literature.  Messages  must  be  displayed  long 
enough  to  provide  meaningful  feedback  (Simpson,  1982:118). 
This  particular  error  message  should  also  have  been 
consistent  with  all  of  the  others  (Menkus ,  1983:13;  Simpson, 
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1982:118;  Shneiderman ,  1987:61).  Other  error  messages  remain 
displayed  much  longer  or  until  the  user  initiates  an  action. 

The  CAPPS  module  employed  system  messages  throughout  the 
program.  The  participants  rated  the  messages  as  to  how  clear 
they  were.  The  mean  participant  rating  was  4  with  a  standard 
deviation  of  1.63.  Figure  12  gives  the  participant  rating 
distribution  and  range. 


Participant  Ratings 

Fig.  12.  CAPPS  System  Message  Histogram 
Designing  Dialogs  to  Yield  Closure .  Three 


participants  made  comments  suggesting  confusion  about  knowing 
whether  a  task  was  complete,  and  if  so,  where  the  program 
would  place  them  after  task  completion.  Two  participants 
commented  about  using  the  "Quit"  option  found  throughout  the 
CAPPS  module.  One  of  the  participants  asked  whether  the 
option  would  get  him  out  of  a  particular  function  or  if  it 
would  end  the  program.  The  other  participant  tried  the  option 
to  end  a  function  and  ended  up  exiting  the  whole  program. 

Three  participants  near  the  end  of  task  four,  had 
problems  determining  if  changes  had  been  saved  to  the 
database.  One  participant  stated  that  his  work  had  not  been 
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saved;  in  reality  it  had,  but  it  was  not  shown.  The 
literature  is  in  agreement  with  all  of  the  participant 
opinions  in  this  section.  Action  sequences  should  have 
definite  starting  and  ending  points  so  users  can  get  a  sense 
of  accomplishment  (Shneiderman ,  1987;62). 

Consistency  in  Sequences  of  Actions .  Participant 
comments  about  consistency,  like  those  with  the  GAT  module, 
centered  around  the  movement  issues  and  the  appropriate  keys 
to  use.  All  comments  about  consistency  were  mentioned  in  the 
section  covering  the  ease  of  movement  throughout  CAPPS. 

Error  Handling .  Aside  from  the  opinion  scale,  only  two 
participants  remarked  further  about  error  handling  in  CAPPS. 
One  participant,  although  he  was  not  specific,  stated  having 
a  problem  with  the  general  way  the  program  allowed  error 
correction.  Another  participant  said  it  was  a  cumbersome 
process  to  correct  errors  and  not  at  all  like  the  programs  he 
frequently  used.  He  further  suggested  allowing  users  the 
ability  to  page  back  to  aid  in  error  correction.  The 
literature  states  error  correction  should  be  made  as  simple 
as  possible  (Shneiderman,  1987:62).  The  page  back  option  also 
suggested  would  permit  easy  reversal  of  actions  which  could 
reduce  user  anxiety  about  errors  (Shneiderman,  1987:62). 

One  area  of  concern  in  error  handling  is  the  prevention 
of  catastrophic  errors.  One  participant  destroyed  the  program 
database  because  he  did  not  notice  the  displayed  warning  that 
occurs  when  you  begin  entering  data  for  a  new  program.  He 
expressed  concern  about  how  easily  the  database  was 
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destroyed.  He  also  suggested  the  program  should  be  modified 
so  if  a  change  is  made,  the  old  database  should  be  kept  for  a 
period  of  time.  This  opinion  is  also  in  agreement  with  the 
literature.  For  a  period  of  time,  all  actions  should  be 
reversible  (Norman,  1983:257).  Auditory  signals,  warning 
colorations,  and  highlighted  messages  can  be  used  to  insure  a 
user  notices  a  warning. 

The  participants  rated  the  CAPPS  module  on  how  easy  and 
time  efficient  it  was  to  correct  errors.  The  mean  participant 
rating  was  4.28  with  a  standard  deviation  of  1.60.  Figure  13 
depicts  the  participant  rating  range  and  distribution. 
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Fig.  13.  CAPPS  Error  Correction  Histogram 


Display  Issues 

Using  Color  for  Highlighting .  Four  participants 
expressed  opinions  about  the  use  of  color  for  highlighting  or 
attention  grabbing  purposes.  Three  participants  said  they  had 
difficulty  distinguishing  menu  items  from  each  other.  To 
avoid  that  type  of  confusion,  they  suggested  using  color  to 
highlight  menu  items.  One  participant  further  stated  he  would 
prefer  coloring  the  option  associated  with  the  cursor 
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different  from  the  others  in  the  menu.  These  opinions  are 
also  backed  by  the  literature  (Rubinstein  and  Hersh, 

1985:163;  Shneiderman,  1987:71).  Besides  these  suggestions, 
reverse  video,  the  use  of  borders,  varying  display  intensity, 
the  use  of  blinking  words,  and  even  putting  more  space 
between  the  options,  are  possible  solutions  to  this  problem. 

Two  participants  stated  the  menu  area  of  the  screen  (see 
Figure  14)  grabbed  their  attention  more  than  the  title  areas. 
This  caused  confusion  by  making  the  participants  think  the 
bracketed  menu  option  was  indeed  the  title.  One  of  the 
participants  suggested  using  colored  titles  to  remedy  this 
problem.  The  comments  at  the  end  of  the  previous  paragraph 
also  apply  in  this  situation. 

Two  participants  commented  that  they  very  much  liked  the 
highlighted  text  areas  for  warning  users  of  impending  project 
problems .  This  agrees  with  the  literature  as  one  of  the  ways 
highlighting  is  used  effectively. 

General  Use  of  Color .  In  rating  the  overall  use  of 
color  for  text  and  background,  one  opinion  was  expressed.  The 
participant  said  users  should  have  color  options  because 
white  lettering  bothered  him.  He  stated  a  preference  for 
yellow  lettering  with  the  dark  blue  background  instead  of  the 
white  lettering.  Again,  consistency  with  other  application 
programs  is  important.  The  literature  also  stresses  the 
importance  of  letting  the  user  control  the  colors 
(Shneiderman,  1987:339). 
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The  CAPPS  graphics  are  designed  to  give  you  a  lot  of  information  in  a 
hurry.  Unfortunately ,  the  sophistication  of  the  analysis  exceeds  the 
capabilities  of  the  computer  to  display  the  data,  so  you  will  probably 
contend  with  a  significant  degradation  of  quality.  There  is  still  ample 
info  available,  however;  and  it  is  immediately  at  your  fingertips. 

Although  CAPPS  was  designed  to  operate  with  a  color  monitor,  various  line 
types  make  operation  with  a  black  and  white  monitor  possible.  The  computer 
must  have  graphics  capability,  however. 


[RETURN]  PREVIOUS_SCREEN  HELP 

Structure  Update  Analyze  Explain 


Fig.  14.  CAPPS  Menu  Display  Problem 


One  question  in  the  opinion  scale  rated  the  participants' 
opinions  as  to  the  use  of  color  for  contrast.  The  mean 
participant  rating  for  character  contrast  with  the  background 
was  4.71  with  a  standard  deviation  of  1.38.  Figure  15  adds 
more  summary  statistics. 


Participant  Ratings 
Fig.  15.  CAPPS  Contrast  Histogram 

The  participants  also  rated  how  appropriate  the  overall 
use  of  color  was.  The  mean  participant  rating  was  5.42  with  a 
standard  deviation  of  .78.  For  the  rating  distribution  and 
high  and  low  ratings,  see  Figure  16. 

Frequency 


Fig.  16.  CAPPS  Overall  Use  of  Color  Histogram 

Graphics .  Three  comments  were  made  about  the 
general  usefulness  of  the  graphical  capabilities  of  CAPPS. 
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One  participant  expressed  a  dislike  for  the  graphics  stating 
the  components  shown  were  hard  to  distinguish  from  each 
other,  so  they  would  not  be  clear  enough  for  overhead 
transparencies  used  at  meetings.  While  the  graphics 
resolution  problem  is  just  an  opinion,  the  researcher  did 
print  out  some  graphics  and  found  that  without  the  color,  the 
resolution  between  components  was  poor.  It  is  important  to 
note  that  two  participants  said  the  graphics  were  fine. 

The  participants  rated  the  helpfulness  of  the  graphics 
provided.  The  mean  participant  rating  for  the  use  of  graphics 
was  5.14  with  a  standard  deviation  of  1.21.  Figure  17  details 
the  participant  opinion  rating  range  and  distribution. 


Participant  Ratings 
Fig.  17.  CAPPS  Graphics  Histogram 

Screen  Layout .  The  neatness  of  the  screen  design 
was  also  rated  by  the  participants.  The  mean  participant 
rating  on  the  overall  screen  display  layout  was  4.71  with  a 
standard  deviation  of  1.38.  For  information  on  rating  range 
and  distribution,  see  Figure  18  below. 
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Participant  Ratings 

Fig.  18.  CAPPS  Screen  Display  Histogram 

Documentation .  Four  opinions  were  expressed  regarding 
documentation.  One  participant  suggested  a  more  detailed 
index  in  the  documentation  would  be  helpful.  He  also  stated 
the  documentation  was  "fine",  but  in  reality  busy  program 
managers  would  not  read  it  and  would  also  prefer  to  have  a 
pamphlet  about  three  pages  long  to  accompany  the  users' 
manual . 

Two  participants  stated  that  some  of  the  online  help  was 
weak.  Both  participants  expressed  the  desire  for  more  online 
instruction  at  points  during  the  program.  The  points  in 
particular  involved  the  overview  function,  or  the  beginning 
screen  and  how  to  end  a  particular  function  without  quitting 
the  program. 

The  participants  rated  online  help  or  documentation  as  t 
its  usefulness.  The  mean  participant  rating  for  online  help 
or  documentation  was  4  with  a  standard  deviation  of  1.82, 


Note  three  participants  chose  to  answer  this  as  not 
applicable  since  they  cLxd  not  use  any  online  help.  Figure  19 
details  range  and  distribution  for  online  help  ratings. 
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Fig.  19.  Online  Help  Histogram 


The  participants  also  rated  the  layout  of  the  written 
documentation  as  to  its  ease  of  use.  They  gave  a  mean  rating 
of  4.42  for  the  general  layout  of  the  documentation.  The 
standard  deviation  was  1.39.  They  also  gave  overall  opinions 
about  the  helpfulness  of  the  documentation.  The  participant 
mean  rating  was  4.57  with  a  standard  deviation  of  1.27.  For 
detailed  information  regarding  the  ratings  of  the  general 
layout  and  helpfulness  of  the  written  documentation,  see 
Figures  20  and  21  below. 
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Participant  Ratings 

Fig.  20.  CAPPS  Written  Documentation  Layout  Histogram 
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Frequency 


Participant  Ratings 

Fig.  21.  CAPPS  Overall  Written  Documentation  Histogram 

Response  Time .  The  only  opinions  expressed  about  the 
system  response  time  were  those  recorded  on  the  written 
opinion  scales.  The  participant  mean  rating  for  the  response 
time  was  5  with  a  standard  deviation  of  1,15.  Figure  22 
details  participant  response  regarding  system  response  time. 

Frequency 


Participant  Ratings 

Fig.  22.  CAPPS  Response  Time  Histogram 

Miscellaneous  Comments .  Two  participants  commented  on 
the  lack  of  a  printing  function  in  the  CAPPS  module  as  is 
provided  in  most  other  programs. 

There  was  a  problem  that  occurred  with  every  CAPPS 
participant.  It  could  be  classified  as  a  combination  display/ 
feedback  problem.  In  the  database  function  after  entering  a 
new  element  name,  the  program  displays  the  new  name  with  a 
question  to  confirm  the  new  name  (see  Figure  23) .  The  program 


displays  the  new  name  and  the  question  without  leaving  a 
space  between  the  two  sentences.  This  made  the  participants 
think  they  had  misspelled  the  new  element  name,  when  in  fact 
they  had  not . 

Three  participants  commented  on  ambiguity  of  terminology. 
Two  participants,  while  trying  to  display  a  graph  of  trends, 
said  the  options  “adjust",  "expand",  and  "zoom"  were 
confusing  (see  Figure  24) .  One  participant  said  none  of  the 
options  was  intuitively  obvious  even  after  finding  the 
correct  one.  Another  participant  said  the  definitions  used  by 
CAPPS  in  the  database  function  (see  Figure  25)  needed  to  be 
more  clear  and  distinguishable. 
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Fig.  23.  CAPPS  Display/ Feedback  Problem 
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Fig.  24.  CAPPS  Terminology  Problem 
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INTERACTIVE  UPDATE 


Add 

Correct 

Ramov* 

Modi f y 

New 

Finished 


Use  this  function  to  add  the  next  period  of  data  to  the 
data  base. 

Use  this  function  to  correct  data  for  a  month  that  is  already 
in  the  data  base. 

Use  this  function  to  remove  (delete)  an  existing  WBS  or 
Functional  element. 

Use  this  function  to  modify  the  contract  name  or  to  add  or 
change  the  contract  completion  date. 

Use  this  function  to  create  a  new  contract  data  base. 

Use  this  function  to  return  to  the  utilities  menu. 


(Add]  Correct  Remove  Modify  New  Finished  Explain 


Fig.  ,25.  CAPPS  Database  Definitions 


V.  Conclusions  and  Reconunendations 


Introduction 

As  in  chapter  four,  H/C  interface  issues  taken  from  the 
literature  review  will  be  used  as  a  format  for  the  body  of 
this  concluding  chapter.  Topics  will  be  discussed  in  the  same 
order  as  in  chapter  four.  An  overall  evaluation  of  the  PMSS 
H/C  interface  will  conclude  this  chapter. 

Each  H/C  interface  issue  discussed  in  chapter  four  will 
be  reviewed  and  conclusions  made  regarding  the  PMSS 
effectiveness  in  those  specific  issues.  Recommendations  for 
improvements  or  further  research  will  be  made  when  warranted. 

General  Issues 

Ease  of  Movement .  Throughout  the  literature,  authors 
have  stressed  consistency  and  have  said  it  is  a  very 
important  part  of  the  H/C  interface.  This  consistency  not 
only  includes  issues  within  the  program,  but  also  issues 
common  to  most  application  programs.  Many  comments  were  made 
in  both  modules  of  the  PMSS  about  the  use  of  the  space  bar 
and  backspace  keys  for  movement.  This  is  a  weak  area  for  the 
PMSS.  The  modules  do  not  consistently  use  one  method  or  the 
other  and  the  participants  were  not  comfortable  with  the  PMSS 
approach.  A  large  number  of  the  participants  in  this  study 
use  spreadsheet  programs  in  their  day  to  day  work  and 
movement  in  spreadsheets  usually  involves  arrow  keys. 

To  make  the  PMSS  consistent  with  other  tools  available 
for  program  managers  and  staff,  the  PMSS  should  be  modified 
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to  allow  movement  using  the  arrow  keys  throughout.  Modifying 
the  PMSS  in  this  manner  could  help  reduce  user  confusion  and 
frustration . 

Consistency  in  Sequences  of  Actions .  Besides  the 
movement  issue  discussed  above,  only  one  negative  comment 
occurred.  The  PMSS  needs  to  be  consistent  in  the  area  of 
saving  work  just  recently  accomplished.  The  database 
functions  save  the  work  after  the  user  exits,  but  in  other 
areas,  the  work  or  changes  are  saved  before  the  user  exits. 
After  taking  care  of  the  way  users  move  about  and  how  actions 
are  closed  out,  this  part  of  the  interface  could  be 
considered  adequate. 

Offer  Informative  Feedback .  Only  a  minor  problem 
occurred  in  this  area.  The  display  time  needs  to  be 
lengthened  for  at  least  one  error  message.  This  area  of  the 
PMSS  H/C  interface  is  adequate. 

Design  Dialogs  to  Yield  Closure .  There  were  a  few  minor 
problems  in  this  area.  Some  participants  had  trouble 
realizing  when  a  functional  task  was  finished.  There  needs  to 
be  more  system  messages  telling  users  when  tasks  are 
complete.  A  distinction  also  needs  to  be  made  whether  or  not 
the  "Quit"  option  means  quit  the  function  or  quit  the 
program.  For  the  most  part,  this  area  of  the  interface  seemed 
adequate . 
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Error  Handl incr 

Correcting  errors  in  the  PMSS  usually  is  not  difficult, 
but  it  is  not  as  easy  as  it  should  be.  The  PMSS  should  be 
modified  so  that  all  modules  have  online  help.  Installing  the 
page  back  option,  like  the  one  in  the  GAT  module,  throughout 
the  PMSS  would  make  error  correction  easier.  The  system 
should  also  allow  reversal  of  actions,  such  as  changes  to  the 
database,  to  prevent  catastrophic  errors.  Some  programs 
provide  a  “Whoops"  option  that  reverses  an  unwanted  action 
that  just  occurred.  This  type  of  option  would  prevent  some 
errors  and  might  lead  to  a  higher  degree  of  user 
satisfaction.  Without  these  changes,  error  handling  in  the 
PMSS  is  satisfactory,  but  as  one  participant  stated,  “a 
cumbersome  process." 

Display  Issues 

Color .  In  the  issue  of  contrast,  the  PMSS  uses  color  in 
an  sufficient  manner.  The  system  uses  default  colors  that  did 
not  trouble  the  participants  greatly,  but  they  did  not  feel 
the  optimal  choice  or  choices  were  used.  The  options 
available  to  the  user  are  either  limited  or  do  not  exist . 
First  of  all,  the  PMSS  should  be  changed  to  allow  color 
contrast  options  in  all  of  the  modules.  Secondly,  users 
should  be  allowed  to  choose  from  a  variety  of  options  or 
devise  their  own. 

In  general,  color  use  in  the  PMSS  is  adequate.  The  PMSS 
does  a  good  job  of  using  color  for  highlighting  or  warning 
purposes.  There  is  again  the  problem  of  color  options.  Again, 
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the  system  should  be  modified  to  allow  color  options  in  all 
modules,  as  is  done  in  most  application  programs.  All  modules 
should  also  allow  the  user  to  turn  off  the  colors.  This  would 
increase  user  satisfaction  by  allowing  the  user  to  modify  the 
program  to  his/her  own  personal  tastes. 

Screen  Layout .  The  layout  of  the  PMSS  was  rated 
favorably  by  the  participants  and  should  be  considered 
adequate.  The  screens  are  generally  uncluttered  and  neat. 

Menu  items  are  clearly  distinguishable  in  the  GAT  module,  but 
cramped  in  the  CAPPS  module.  The  menu  item  display  needs  to 
be  made  consistent  with  the  choices  clearly  distinguishable. 
When  editing  text,  the  user  should  decide  what  is  removed, 
instead  of  reaccomplishing  the  whole  field.  Modifying  the 
PMSS  in  this  manner  would  make  the  system  consistent  with 
other  application  programs  available  in  the  Air  Force. 

Graphics .  The  ability  of  the  PMSS  to  display  graphics  is 
good.  This  conclusion  is  based  on  the  CAPPS  module  only 
because  the  GAT  module  did  not  use  graphics.  More  research 
needs  to  be  done  with  PMSS  graphics.  This  is  an  area  the 
participants  did  not  have  to  interpret  during  the  scenarios. 
The  participants  just  rated  the  general  look  and  not  the 
functionality  of  the  graphics.  The  resolution  of  the  graphics 
in  some  types  of  monitors  and  on  paper  may  not  be  adequate. 

Documentation 

The  written  documentation  provided  with  the  PMSS  is  good. 
The  approach  of  what  to  do,  how  to  do  it,  and  why  do  it,  is 
effective  for  novice  and  intermittent  users.  The 
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documentation  also  uses  screen  representations  in  a  way  that 
is  also  helpful.  The  only  modification  this  study  recommends 
is  the  addition  of  a  detailed  index  to  aid  users  in  searches. 

The  online  documentation  provided  by  the  PMSS  is  somewhat 
less  than  adequate.  The  participants  did  not  use  the  online 
documentation,  but  it  was  reviewed  by  the  researcher  and 
project  officer.  Messages  or  text  displayed  on  the  screen 
should  be  professional  looking  and  should  not  try  to  be 
humorous  (see  Figure  26) .  The  online  documentation  should  be 
modified  so  all  words  are  spelled  correctly  and  attempts  at 
humor  reviewed  carefully. 

Response  Time 

This  area  of  the  PMSS  H/C  interface  was  rated  very 
favorably  by  the  participants  and  is  comparable  to  most 
application  programs.  Some  modules  of  the  R1SS  are  more 
complex  than  others,  but  the  response  time  should  be  no 
problem  if  the  programs  are  on  the  microcomputer  hard  drive. 

Conclusion 

There  is  one  major  problem  that  is  easily  seen  in  the 
future  for  the  PMSS,  and  that  problem  is  consistency.  The 
modules  used  in  this  and  a  parallel  study,  that  will 
eventually  be  combined  to  form  the  PMSS,  have  significant 
differences  in  the  many  areas  of  the  H/C  interface.  Some  of 
the  obvious  differences  include  how  color  is  used,  the  style 
of  written  documentation,  the  use  and  quality  of  graphics, 
the  availability  of  online  help,  the  way  users  move 
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throughout  the  programs,  and  even  the  basic  interaction 
styles.  The  literature  very  strongly  suggests  that  if 
consistency  in  the  H/C  interface  is  not  maintained,  users 
will  not  readily  accept  the  PMSS. 

Most  of  the  issues  of  the  H/C  interface  were  handled 
adequately  when  these  particular  modules  of  the  PMSS  were 
developed.  If  the  recommended  modifications  are  accomplished, 
the  system  should  be  accepted  very  favorably  in  the  user 
community . 
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HARDCOPY 


There  is  a  standard  feature  on  most  micro  computers  that  allows  you  to  get 
a  "screen  dump"  from  you  CRT.  CAPPS  takes  advantage  of  this  feature,  so  as 
long  as  you  have  a  printer  PROPERLY  CONFIGURE  FOR  YOU  MACHINE,  you  should 
have  no  problem  getting  hardcopy  printouts  of  anything  your  little  old 
heart  desires.  Just  remember,  it  will  only  occur  at  the  speed  of  your 
printer,  so  don't  blame  CAPPS  if  you  have  to  wait  a  minute  or  two!! 
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Appendix  A 
GAT  Scenario 


GOVERNMENT  ACTIVITY  TASKING  (GAT)  SCENARIO 

In  this  scenario,  you  will  be  using  the  GAT  module  of  the 
Program  Manager's  Support  System  (PMSS) .  You  will  be  asked  to 
do  tasks  that  might  be  found  in  your  current  job  situation  as 
well  as  those  tasks  that  GAT  was  designed  to  accomplish. 

Task  #1  -  Edit  an  existing  activity.  There  has  been  a 
change  in  ECP  0011.  Change  the  activity  to  reflect  the 
foil  owing : 

1.  Appropriation  =  2257 

2.  Required  Completion  Date  =  Jul  90 

3.  Milestone  S2  should  read  "Transfer  Funds  to 
Sacramento  ALC . " 

4.  Tasking  Organization  Manager  =  Major  Burns 

5.  Total  FY  Funding  =  9.0 

6 .  Labor  =  4.0 

7.  1st  Quarter  Indirect  Support  =  .26 

8.  4th  Quarter  Other  Government  =  .35 

Task  ^2  -  Print  the  GAT  Worksheet  Summary  for  ECP  0006 
(LRP  Modification)  . 

Task  #3  -  Create,  Save,  and  Print  a  report  with  the 
following  headers  in  the  order  they  are  listed  below.  Save 
the  report  under  any  name  you  desire. 

1.  Program  Title 

2.  Required  Completion  Date 

3.  Fiscal  Year 

4 .  Labor 

5.  Funding  Totals 
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Appendix  B 
CAPPS  Scenario 


CONTRACT  APPRAISAL  SYSTEM  CAPPS  SCENARIO 

You  are  currently  working  in  a  system  program  office 
that  is  well  into  development  of  a  new  attack  aircraft  only 
known  as  the  Long  Range  Penetrator  (LRP) .  The  program  was 
started  in  August  1983  and  is  due  to  be  completed  in  June 
1986.  The  current  month  of  development  is  July  1984  and  the 
program  office  uses  the  Program  Manager's  Support  System  as 
a  tool  to  track  contract  status. 

Task  #1 

A.  Print  the  status  report  of  the  element  with  the 
largest  negative  cost  variance. 

B.  Print  a  graphical  chart  that  shows  the  variance 
trends  of  the  element  with  the  largest  negative  cost 
variance . 

C.  Print  a  graphical  chart  that  shows  the  current 
estimates  at  completion  for  that  same  element. 


Task  #2 

A.  Print  the  PMSS  status  report  of  element  1.6  (System 
T  &  E)  . 

B.  Print  a  graphical  chart  that  shows  the  variance 
trends  for  element  1.6  over  the  past  six  months. 

Ta-k  #3 

A  contractor  makes  an  error  reported  in  element  1.1.3 
ACWP  for  March  1984.  The  value  should  read  3000.  Change  the 
data  base  to  reflect  the  correct  figure. 

Task  #4 

A  new  element  has  been  added  to  the  LRP.  It  falls  under 
element  1.2  Carrier  Acrft .  Integrity.  Add  the  new  element  to 
the  data  base  using  the  following  information: 

A.  Element  Name  =  1.2.1  Software  Integration 

B.  Start  Date  =  June  1984 

C.  BCWS  =  4000 

D.  BCWP  =  4050 

E.  ACWP  =  4500 

F.  BAC  »  8000 

G.  LRE  =  8000 
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Appendix  C 


PMSS  Briefing  Outline 

General  Overview  for  Intro  to  PMSS  Briefing 

-Introduction  of  individuals  involved 
— Two  AFIT  students 
—Mr  Miller  ASD/XRI 
— Anyone  else  playing  a  part 

-State  the  purpose  of  the  briefing 

— Explanation  of  tasking  (both  ASD' s  and  AFIT  thesis) 

— Introduction  to  the  PMSS 

— Introduction  to  our  PMSS  testing  procedures 
— Setup  of  tentative  interview  calendar 

-Explain  the  reasons  for  doing  this  research 
— ASD  reasons 

- Possible  future  use  as  a  Decision  Support  System 

- Inputs  from  research  can  change  the  prototype 

- Inputs  can  influence  the  decision  to  use  the  PMSS 

— AFIT  reasons 

- Fulfill  thesis  requirements 

-Introduction  of  the  PMSS 
— Developer 

— Purpose  of  the  system 
— Structure  of  the  system 
— ID  specific  modules 

- Explain  the  purpose  of  each 

-Explain  testing  procedures 

— Stress  the  system  is  being  tested  and  not  the  user 
— Stress  anonymity  of  the  interviewee 
— Explain  protocol  collection  concept 
--Explain  the  use  of  recording  devices 
— Explain  documentation  and  scenario  preview 
— Explain  debrief  at  the  end  of  interview 

-Q  &  A 

-Setup  of  tentative  interview  calendar 
-Concluding  remarks 
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Appendix  D 

Test  Area  Layout 


Extra  DMk 


Appendix  E 


Agreement  Statement 


STATEMENT  OF  AGREEMENT 

In  order  to  collect  your  open  responses  and  therefore 
the  best  possible  data  in  this  research  effort,  I  want  to 
again  assure  you  of  your  anonymity.  Any  comments  you  give  me 
while  participating  in  this  study  will  not  be  attributed  to 
you  in  any  way. 


Kelly  R.  Fulcher,  ILT,  USAF 
PMSS  Researcher 


I  understand  my  freedom  to  comment  on  any  aspect  of  this 
research  effort. 


PMSS  Research  Participant 
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Appendix  F 
Game  Details 

The  games  used  to  warm  up  participants  is  called  NYET . 

It  was  written  by  David  B.  Howorth  in  1988  and  is  considered 
part  of  Public  Domain  software. 

The  game  in  its  use  and  screen  layout  demonstrates  many 
areas  of  the  H/C  interface.  Graphics,  color,  text  and  numeral 
display,  and  etc.  The  object  of  the  game  is  to  fit  falling 
shapes  into  a  box,  leaving  as  little  blank  space  as  possible. 
The  speed  is  variable  and  points  are  accumulated  by  the 
number  of  falling  shapes  the  user  handles  and  by  filling  rows 
with  shapes . 
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Appendix  G 
Opinion  Scale 


1 .  Character  contrast  with  background 
poor/unreadable  0123456  excellent/readable 

2.  Overall  use  of  color 

inappropriate  0123456  appropriate 

3.  Online  help  facilities 

not  useful  0123456  N/A  very  useful 

4 .  System  feedback  or  messages 
confusing  0123456  clear 

5 .  Overall  screen  display  layout 
cluttered/messy  0123456  uncluttered/neat 

6.  Overall  use  of  graphics 

not  helpful  0123456  N/A  helpful 

7 .  Ability  to  correct  errors 

hard/uses  time  0123456  easy/time  efficient 

8.  Overall  ease  of  movement  throughout  the  module 
hard/uses  time  0123456  easy/time  efficient 

9.  Layout  of  %n:itten  documentation 

hard  to  use  0123456  easy  to  use 

10.  Overall  opinion  of  written  documentation 
not  helpful  0123456  helpful 

11.  Time  it  takes  the  system  to  respond  to  commands 
too  long  0123456  just  right 
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Appendix  H 


Concluding  Questions 

1.  Have  you  had  any  formal  computer  education? 

2.  Do  you  use  a  computer  on  the  job? 

3.  Approximately  how  much  time  did  you  spend  viewing  the 
documentation? 

4.  Is  this  module  something  you  could  use  on  the  job? 

5.  Do  you  have  any  concluding  remarks? 
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Appendix  1 


GAT  Summary  Statistics 


Question 

Mean 

sp 

L  .  Value  S . 

Value 

#1 

5.142 

.899 

6 

4 

#2 

5.428 

.786 

6 

4 

#3 

Not 

Applicable  to 

GAT 

#4 

4.285 

1.380 

6 

3 

#5 

5.285 

.755 

6 

4 

#6 

Not 

Applicable  to 

GAT 

#7 

4  .285 

1.496 

6 

2 

#8 

4.571 

1.902 

6 

2 

#9 

4.714 

1.380 

6 

3 

#10 

5.142 

.690 

6 

4 

#11 

5.428 

.786 

6 

4 

Obs  . 

7 

7 

7 

7 

7 

7 

7 

7 

7 


Outliers 
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Appendix  J 

CAPPS  Summary  Statistics 


Question. 

Mean 

#1 

4  .714 

#2 

5.428 

#3 

4.000 

#4 

4.000 

#5 

4.714 

#6 

5.142 

#7 

4.285 

#8 

3.428 

#9 

4  .428 

#10 

4.571 

#11 

5.000 

^  L.  Value 

1.380  6 

.786  6 

1.825  6 

1.632  6 

1.380  6 

1.214  6 

1.603  6 

.975  5 

1.397  6 

1.272  6 

1.154  6 


S.  Value  Obs . 


2  7 
4  7 
2  4 

2  7 

3  7 
3  7 
2  7 

2  7 

3  7 
3  7 
3  7 


Outliers 
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Appendix  K 
GAT  transcriptions 


Transcription  GAT  A 
Task  1 


Nar 

Rschr 

Resp 

Nar 


Resp 


Nar 

Resp 


The  introductory  screen  is  displayed. 

What  do  you  think  of  the  colors  used? 

This  looks  comfortable,  (default  colors) 

The  resp.  edits  the  data  as  per  scenario 
instruction . 

One  thing  I  would  like  to  see  on  these  things  is 
when  you  have  to  scroll  programs  like  this,  going 
from  left  to  right,  it  takes  alot  of  time.  I  don't 
know  why  they  don't  set  them  up  so  you  can  scroll 
down  one  side  and  go  over  if  you  need  to.  That  would 
be  more  efficient. 

The  resp.  finishes  the  task. 

Well,  I've  done  everything  required  for  task  1  and 
as  I  said,  the  only  thing  I  would  change  is  the  way 
you  scroll  around  the  page. 


Task  2 

Nar  -  The  GAT  worksheet  summary  page  is  displayed,  which 
is  where  the  last  task  ended.  The  resp.  completes 
the  task  without  further  comment . 


Task  3 


Nar 


Resp 


Nar 

Resp 

Rschr 

Resp 

Rschr 

Nar 

Resp 


The  GAT  worksheet  summary  page  is  displayed,  which 
is  where  the  last  task  ended.  The  resv .  begins  to 
create  the  report  as  per  scenario  instruction.  The 
resp.  made  an  error  of  creating  a  new  ECP  and  then 
proceeded  on  the  right  path  to  create  the  report . 
It  would  be  nice  if  these  (headers)  were  in 
alphabetical  order.  They  would  be  alot  easier  to 
find . 

The  resp.  is  about  halfway  through  the  task. 

Is  this  going  to  print  the  complete  title? 

Yes  . 

It  is  not  going  to  drop  the  title  and  end  off. 

No,  it  prints  it  all  even  if  it  cuts  it  off  on  the 
SCI ^en . 

The  resp.  finishes  the  task. 

Say  you're  modifying  something  and  your  eyes  get 
tired,  all  you  have  to  do  is  go  back  to  the  main 
screen  with  the  FIO  key  and  you  can  change  colors. 
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Rschr  -  Yes . 

Resp  -  Some  of  our  programs  can't  change  and  that  drives 
you  crazy  after  awhile. 


Scale 


No  Expanding  Comments 


Concluding  Remarks 


Q  #1 
Q  #2 


Q  #3 
Q  #4 
Q  #5 


I  took  some  college  courses  in  programming  Fortran. 
Yes,  all  of  the  project  officers  use  the  computer 
alot .  It  is  mostly  used  for  word  processing  and 
making  charts. 

About  a  half  and  hour. 

We  don’t  do  this,  our  finance  office  does  it. 
Sometimes  I  question  the  use  of  fancy  software. 

I  do  prefer  a  dark  blue  background  with  cyan,  white, 
or  pinkish  lettering. 


Transcription  GAT  B 
Task  1 


Nar 

Resp 

Rschr 

Resp 


Nar 

Resp 


The  introductory  screen  is  displayed.  The  resp, 
edits  per  scenario  instruction. 

You  just  enter  to  move  on  down.  I  guess  the  arrow 
keys  can  be  used  also,  but  they  don't  let  you  edit 
that  way. 

Do  you  think  they  should  use  the  arrow  keys  for 
editing. 

I  guess  you  can  look  at  it  either  way.  On  the  first 
screen  it  didn't  allow  you  to  move  with  the  arrow 
keys  and  on  the  second  it  does .  They  need  to  make  it 
consistent  one  way  or  the  other.  I  figured  you  could 
move  in  the  text  with  the  arrow  key  because  the 
first  screen  wouldn't  allow  it.  Some  programs  allow 
you  to  move  between  blocks  with  the  tab  key  and  then 
let  you  use  the  arrow  key  within  for  editing. 

The  resp.  nears  finishing  the  task. 

The  editing  ought  to  start  over  on  the  forward  right 
edge  like  a  calculator  and  force  the  number  out  as 
it  goes .  That  way  you  would  know  you  are  typing  over 
it  . 


Task  2 

Nar  -  The  summary  screen  is  displayed  and  that  happens  to 
be  where  the  last  task  ended.  The  resp.  completes 
the  task  without  further  comment . 


83 


Task  3 


Nar  -  The  summary  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  proceeds  to  create  the 
report  as  per  scenario  instruction. 

Resp  -  I'm  not  sure  how  they  determined  the  order  of  those 
(headers) ,  but  I  think  it  would  be  better  if  they 
were  alphabetized  or  in  at  least  some  kind  of  order. 

Nar  -  The  resp.  finishes  the  task  without  further  comment. 


Scale 


Q  #1 

Q  #2 

Q  #4 

Q  #7 

Q  #8 
Q  #9 

Q#10 


Character  contrast  was  OK,  but  I  usually  like  the 
dark  blue  background  with  the  light  blue  writing.  It 
is  not  quite  as  stark  a  contrast  as  white  and  is 
more  readable  than  grey. 

I  like  the  dark  blue  background  the  best  of  all  the 
choices.  Overall  use  of  color  is  fine.  I  don’t  see 
any  problem  with  it. 

Most  of  the  queries  are  pretty  self  explanatory 
except  where  it  asked  you  if  you  want  to  save  the 
changes.  The  way  I've  seen  other  programs  do  it 
is...  they  will  ask  if  you  really  want  to  quit  and 
then  ask  you  if  you  want  to  save  the  changes . 

Ability  to  correct  errors  leaves  something  to  be 
desired.  When  you  start  typing,  blank  it  and  then 
put  on  the  new  numbers  or  have  a  little  window  and 
when  you  fill  it,  it  automatically  changes  it. 

The  module  is  pretty  easy  to  move  through. 

The  written  documentation  was  pretty  decent,  it  just 
seemed  to  be  alot  longer  than  it  needed  to  be, 
considering  how  simple  this  thing  is  to  use.  I  don't 
know,  it  may  be  necessary. 

It  was  OK.  It  certainly  wasn't  bad,  because  after  I 
read  it  I  felt  pretty  confident  that  I  could  go  in 
and  after  a  little  playing  around,  I  could  use  the 
thing . 


Concluding  Remarks 

Q  #1  -  I've  had  basic  Fortran,  microprocessor  programming, 
and  machine  language  programming. 

Q  #2  -  Every  day  and  all  the  time.  I  use  it  for  word 

processing,  spreadsheets,  schedule  tracking,  desktop 
publishing,  and  briefing  slides. 

Q  #3  -  About  one  half  an  hour. 

Q  #4  -  Relevant  in  some  cases. 

Q  #5  -  Like  most  government  developed  software,  the 

function  seems  to  be  more  important  than  the  user 
interface.  It  is  not  as  user  friendly  as  most  but 
not  all  commercial  software.  This  one  is  simple 
enough  that  it  doesn't  make  mush  difference. 


84 


Transcription  GAT  C 


Task  1 


Nar 

Resp 

Rschr 

Resp 

Nar 

Resp 

Rschr 

Resp 

Rschr 

Resp 

Nar 


Tne  introductory  screen  is  displayed.  The  resp. 
begins  to  work  the  task. 

Does  this  thing  go  up  (cursor)? 

What  did  you  hit? 

The  up  arrow  key. 

The  resp.  figures  the  problem  out  using  other  keys 
and  continues . 

It  won't  take  that  75  out  of  there. 

Hit  return. 

Oh.  .  . 

Is  that  different  from  what  your  used  to. 

The  space  bar  would  normally  take  that  out  and  I 
would  continue  on  typing. 

The  resp.  continues  and  finishes  the  task. 


Task  2 

Nar  -  The  summary  page  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  tries  the  color  switch 
and  leaves  it  on  the  default.  The  resp.  then 
finishes  the  task. 


Task  3 


Nar 


Resp 

Rschr 

Resp 

Nar 


The  summary  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  works  the  task  as  per 
scenario  instruction. 

Oh.  .  . 

What  did  you  do? 

I  hit  F4  to  try  and  load  it  (header)  into  the  box. 
The  response  was  incorrect  and  then  the  resp. 
tries  to  type  the  header  into  the  box,  again  without 
success.  The  resp.  Then  hits  return,  and  receives  a 
message  stating  it  was  an  invalid  response.  The 
resp.  then  tries  to  edit  and  insert  the  headers, 
which  is  again  incorrect.  The  resp.  then  looks  in 
the  documentation  and  finds  the  correct  header  entry 
method.  The  resp.  then  finishes  the  task. 


Scale 

Q  #1  -  That  was  OK. 

Q  #3  -  They  were  clear. 
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Concluding  Remarks 

Q  #1  -  Intro  to  computers  for  BA  degree  and  OJT  for  DBase. 
Q  #2  -  Not  much,  just  word  processing  mostly. 

Q  #3  -  5  minutes . 

Q  #4  This  is  relevant  to  my  job. 


Transcription  GAT  D 
Task  1 


Nar 

Resp 

Rschr 

Resp 


Nar 

Resp 


Nar 

Resp 


Nar 


The  introductory  screen  is  displayed  and  the  resp. 
begins  to  work  the  scenario. 

Lets  see. . .  How  do  you  move  around?  Oh. . .  Only  the 
right  and  left  arrow  keys  work. 

Do  you  have  a  problem  with  that? 

No  I  don't,  it  is  just  different  from  what  I’m  used 
to  doing.  Does  it  have  a  home  key  to  take  it  to  the 
top? 

The  resp.  tries  the  backspace  key  and  meets  success. 
Oh...  OK,  I  don't  have  a  problem  with  that.  I  would 
have  to  use  it  a  couple  of  times  as  an  adjustment  to 
get  away  from  the  arrow  key. 

The  resp.  continues  to  work  the  task. 

Does  it  file  in  Caps.?  I've  got  some  programs  that 
it  doesn't  matter  how  you  type  it,  it  stores  it  in 
Caps.  I  did  it  that  way  because  I'm  used  to  that. 

The  resp.  finishes  the  task. 


Task  2 

Nar  -  The  summary  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  finishes  the  task  without 
comment . 


Task .3 

Nar  -  The  summary  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  begins  the  task  as  per 
scenario  instruction. 

Resp  -  I  like  the  option  where  you  just  run  the  cursor  over 
it  (header)  and  highlight  it,  and  then  hit  the  enter 
key  and  drop  it  down.  1  think  that  is  a  good 
feature.  In  this  I  also  like  the  option  of  using  all 
the  arrow  keys  to  go  left,  right,  up,  or  down. 

Nar  -  The  resp.  then  finishes  the  task. 
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Scale 


Q  #1  -  1  did  like  the  character  contrast  with  the 

background  and  I  did  like  the  options  available  to 
change  that . 

Q  #2  -•  The  was  great,  although  if  you  had  a  system  that 
didn't  have  color,  1  don't  think  it  would  be  a 
negative . 

Q  #7  -  That  would  be  very  easy . 

Q  #8  -  Very  good. 

Q  #9  -  I  like  that  users  manual.  I  like  the  step  through 
sequence . 

Q#ll  -  Since  this  is  with  a  disk  and  not  a  hard  drive,  I 
think  the  system  response  is  excellent . 


Concluding  Remarks 

Q  #1  -  I've  had  some  basic  OJT  courses. 

Q  #2  -  I  use  the  computer  every  day  for  monitoring  and 
creating  schedules.  I  also  use  them  for  briefing 
slides . 

Q  #3  -  10  minutes. 

Q  #4  -  The  module  is  50%  relevant  to  my  job. 


Transcription  GAT  E 
Task  1 


Nar 

Resp 

Nar 

Rschr 

Resp 

Rschr 

Resp 

Rschr 

Resp 


Rschr 

Resp 


Nar 


The  introductory  screen  is  displayed.  The  resp. 
tries  the  color  toggle  switch. 

I  like  this  best  (default  option)  . 

The  resp.  is  working  on  editing  as  per  scenario 
instruction . 

Why  did  you  hit  the  space  bar? 

I  hit  the  space  bar  so  the  cursor  would  go  over. 

What  did  it  do? 

It  just  stayed  there.  It  printed  out  the  two  in 
white  letters  instead  of  the  grey  letters. 

So  you  were  expecting  the  grey  letters? 

I  was  expecting  it  to  cover  the  grey  letters.  Say 
right  here  on  1.5,  I  tried  to  put  2  and  press  the 
space  bar,  and  it  wouldn't  go  over  it. 

What  do  you  think  about  that? 

I've  used  programs  before  that  you  could  hit  the 
space  bar  and  so  in  the  block  your  using  if  you  hit 
the  space  bar,  it  goes  over  two  spaces  like  I  wanted 
it  to.  Now  I  have  to  go  back  and  change  it. 

The  resp.  continues  with  the  task. 


Rschr 

Resp 


Rschr 


Resp 

Nar 

Resp 


Are  you  wondering  why  that  90  stayed  there? 

Yes,  but  then  i  realized  if  your  going  to  change  the 
block,  you  have  to  change  the  whole  block.  It  takes 
a  little  bit  of  experimenting  to  realize  that. 

Would  you  prefer  they  left  you  the  option  of 
changing  what  you  want  to  change  and  leaving  the 
rest? 

It  might  make  it  a  little  easier.  I  would  prefer  it. 
The  resp.  changes  a  milestone  title. 

I'd  like  to  keep  some  of  the  stuff  in  there  if  I 
could,  but  when  I  go  it  will  erase  everything. 


Task  2 


Nar  -  The  summary  screen  is  displayed.  The  resp.  completes 
the  task  without  further  comment . 


Task  3 

Nar  -  The  summary  screen  is  displayed.  The  resp  has  some 
difficulty  accomplishing  this  task,  but  the  only 
comments  made  during  that  effort  are  as  follows. 
Resp  -  Why  can’t  you  just  type  them  (headers)  out? 


Scale 


Q  #1 

Q  #4 
Q  #5 

<3  #7 

Q#10 

Q#ll 


They  only  give  you  three  color  choices.  I  know  the 
Z-248  has  16  different  color  bands. 

System  feedback  was  good. 

The  overall  screen  display  was  neat,  very  orderly, 
and  a  good  job. 

You  had  to  realize  where  you  were  at  to  correct 
errors . 

I  thought  the  documentation  was  fairly  easy  to  use. 
This  is  much  easier  than  the  usual  Z-248 
documentation . 

Perfect,  right  away. 


Concluding  Remarks 

Q  #1  -  One  computer  science  course.  It  was  an  all 

encompassing  introduction  to  computer  science 
course . 

Q  #2  -  Yes,  mostly  for  word  processing. 

Q  #3  -  Just  for  a  few  minutes. 

Q  #4  -  Yes,  we  do  ECP  work. 


I 
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Transcription  GAT  F 
Task  1 


Nar 


Rschr 

Resp 

Nar 

Resp 

Rschr 

Resp 

Nar 

Rschr 

Resp 


The  introductory  screen  is  displayed.  The  resp. 
edits  as  per  scenario  instruction.  The  resp.  tries 
to  use  the  up  and  down  arrow  keys  to  move  and  can't. 
Would  you  prefer  being  able  to  use  the  up  and  down 
arrow  keys  also? 

Yes  . 

The  resp.  continues  on  the  editing  process. 

How  do  you  move  the  edit  cursor  to  the  right? 

You  have  to  type  over  it. 

Well...  I  don't  like  that. 

The  resp.  is  trying  to  edit  the  funding  niui\bers . 

Is  it  strange  having  the  previous  numbers  stay 
there? 

Yes,  it  is  a  little  confusing.  I  guess  it  is  OK 
after  the  first  time  you  go  through  it.  For  somebody 
that  is  not  an  experienced  user,  it  could  be  a 
little  confusing. 


Task  2 


Nar  -  The  summary  screen  is  displayed.  The  resp.  takes 
this  time  to  try  the  color  toggle  switch  before 
finishing  the  task. 

Resp  -  I  would  not  choose  any  other  color  than  the  default 
option.  I  like  a  dark  blue  background  with  light 
blue  lettering  the  best. 


Task  3 


Nar 


Resp 

Rschr 

Resp 

Rschr 

Resp 


Rschr 

Resp 

Nar 

Rschr 

Resp 


The  summary  screen  is  displayed.  The  resp.  has  no 
problems  with  entering  headers  and  begins  to  finish 
the  task. 

It  says  the  field  size  and  length  is  so  and  this 
header  is  much  longer  than  that. 

It  prints  it  all,  but  it  cuts  it  off  on  the  screen. 
It  will  truncate  it? 

Any  problem  with  that? 

Mrom...  I  guess  with  other  databases  it  is  fairly 
normal ,  so  probably  after  you  have  been  through  it 
one  time,  you'll  be  OK.  Are  these  (headers)  on  the 
screen  in  any  particular  order? 

I  don't  think  so. 

It  would  help  in  finding  them  if  they  were  in 
alphabetical  order. 

The  resp.  finishes  the  task  and  tries  to  save  the 
report  format,  but  fails  to  do  so. 

What  did  you  hit? 

FIO  to  exit. 
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Rschr  -  There  was  a  save  option.  Maybe  it  didn't  show  up 
very  well . 

Resp  -  Obviously  I  missed  it,  but  I  think  I  was  programmed 
from  the  previous  task  to  exit  before  I  could  save. 


Scale 


No  Expanding  Comments 


Concluding  Remarks 

Q  #1  -  I've  done  a  little  basic.  I'm  not  an  expert,  I  have 
just  played  around  with  microcomputers  at  home. 

Q  #2  -  Word  processing,  spreadsheets,  and  viewgraphs. 

Q  #3  -  35-40  minutes. 

Q  #4  -  I  would  probably  use  it  if  I  had  to  manage  several 
projects  in  a  program. 

Q  #5  -  I  don't  see  much  wrong  with  it.  It  is  pretty  good 
really...  Just  a  couple  of  minor  problems. 


Transcription  GAT  G 
Task  1 


Nar 

Resp 

Rschr 

Resp 

Rschr 

Resp 

Nar 

Resp 

Rschr 

Nar 


Rschr 

Resp 

Rschr 

Resp 

Rschr 

Resp 


The  introductory  screen  is  displayed.  The  resp. 
begins  to  perform  the  task. 

Is  "return"  the  only  way  you  can  reverse  your 
fields? 

Try  it.  (resp.  does)  Does  that  bother  you? 

You  just  can't  go  back. 

It  is  not  as  direct  as  an  up  or  down  key. 

I  just  didn't  realize. 

The  resp.  continues  the  task  and  now  needs  to  change 
some  numbers  in  the  scenario  ECP. 

Are  these  fields  formatted  for  decimal  places? 

I'm  sure  they  are  formatted  someway. 

The  resp.  comes  to  the  point  where  the  scenario  asks 
the  resp.  to  change  a  milestone  name.  The  resp. 
attempts  to  delete  a  part  and  encounters  a  problem. 
What  did  you  try  to  do? 

I  tried  to  delete  a  word. 

With  the  delete  key? 

Yes  . 

What  do  you  think  about  that? 

I  should  be  able  to  go  in  and  move  characters  around 
instead  of  typing  over  everything.  You  can't  space 
with  the  cursor  either,  it  just  jumps  to  the  next 
field. 
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Task  2 


Nar  -  The  summary  screen  is  displayed.  The  resp.  works  the 
task  without  problem  or  comment . 


Task  3 


Nar 


Rschr 

Resp 

Rschr 

Resp 

Nar 


The  summary  screen  is  displayed.  The  resp.  works  at 
making  the  report  as  per  scenario  instruction.  The 
resp.  had  considerable  trouble  getting  the  headers 
to  appear  in  the  lower  box. 

What  key  did  you  hit  to  get  that? 

F4  . 

What  did  you  try  after  F4? 

I  just  tried  typing  the  header  name  in. 

The  resp.  finishes  the  task  without  further  comment. 


Scale 

Q  #8  -  Normally,  packages  use  a  tab  key  to  shift  from  box 
to  box.  It  seemed  like  you  had  to  go  a  long  way.  It 
seems  like  you  could  use  a  shift  or  back  tab. 


Concluding  Remarks 

Q  #1  -  Alot  of  classes  on  a  couple  of  programming 
languages.  Nothing  done  on  a  PC. 

Q  #2  -  Yes,  for  charts,  graphics,  and  networking,  and  £- 
mail , 

Q  #3  -  About  an  hour. 

Q  #4  -  It  has  possibilities.  It  would  have  some  benefits 
for  what  I  do. 


Appendix  L 
CAPPS  Transcriptions 


Transcription  CAPPS  A 
Task  lA 

Nar  -  The  resp.  is  viewing  the  overview  screen  at  the 
beginning  of  the  program.  The  resp.  chooses  the 
"all  elements"  option.  The  resp.  moves  to  the 
element  in  question  and  presses  return. 

Resp  -  This  is  what  I  print.  What  does  the  highlighting 
mean? 

Rschr  -  It  is  a  warning  that  something  isn't  right  with 
this  element . 


Task  IB 


Nar 


Resp 


Rschr 

Nar 

Resp 

Rschr 

Resp 


Rschr 

Resp 

Nar 


The  resp.  is  at  the  point  where  the  status  report  is 
showing  for  the  element  in  question.  The  resp. 
chooses  the  "overview"  option  from  the  menu.  This 
choice  puts  the  resp.  back  to  where  he  was  before. 

I  thought  that  overview  was  going  to  tell  me 
something  more  about  this  element...  or  explain 
would  do  that  too. 

It  always  takes  you  back  to  the  beginning  screen. 

The  resp.  then  tries  the  "cumulative  performance" 
option,  which  is  on  the  correct  path. 

The  graphics  is  hard  to  distinguish. 

You  don’t  think  that  is  very  clear? 

No,  not  at  all.  It  doesn't  jump  out  and  grab  me... 
it  is  hard  to  distinguish.  You  should  have  a 
different  scale  on  this  that  is  more  separable.  If 
you  were  going  to  use  these  graphics,  it  would  never 
work.  You  couldn't  tell  the  difference  between  these 
lines  if  they  were  projected  on  a  screen. 

Now  you  need  to  find  the  variance  trends,  because 
you  are  still  on  cumulative  performance. 

Oh...  I'm  used  to  a  cursor  showing  me  where  I'm  at. 
The  resp.  then  finishes  the  task  by  locating  the 
variance  trends  on  the  next  keystroke. 


Task  1C 

Nar  -  The  resp.  is  at  the  point  where  the  last  task  left 
off.  The  variance  trends  screen  is  displayed.  The 
resp.  then  chooses  the  "indices"  option  that  leads 
to  his  correct  choice  of  the  "EAC"  option.  The  EAC 
screen  is  displayed. 
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Resp  -  The  lettering  ought  to  be  from  left  to  right  (the 
screen  shows  the  EAC  lettering  as  up  and  down) . 
Rschr  -  Anything  else  you  want  to  say  about  this? 

Resp  -  Just  the  same  point  about  the  graphics. 


Task  2 A 

Nar  -  The  resp.  is  at  the  point  where  the  last  task  ended. 
The  EAC  screen  is  displayed.  The  resp.  gets  off 
track  for  a  couple  of  minutes  and  finds  the  correct 
element  to  begin  the  task.  The  resp,  the  chooses  the 
"status"  option  and  accomplishes  the  task. 


Task  2B 


Ne-r  -  The  resp.  is  at  the  status  screen,  which  is  where 
the  last  task  ended.  The  resp.  chooses  the  correct 
option  and  gets  to  the  variance  trends  screen. 

Resp  -  Zoom  or  adjust?  (chooses  adjust,  which  is  not 
correct)  It  is  not  status  or  explain. 

Nar  -  The  resp.  chooses  the  "key  element"  option  and  it 
exits  out  of  the  element  being  worked  on. 

Resp  -  Start  that  one  over  again  huh... 

Nar  -  The  resp.  gets  back  to  the  correct  element  by 

spacing  across  the  key  element  screen  to  the  element 
in  question.  The  resp.  then  chooses  the  "zoom" 
option  and  ends  the  task. 

Resp  -  When  I  saw  zoom,  I  thought  that  it  would  zoom  in  on 
one  particular  piece  instead  of  changing  the  scale. 
The  zoom  and  adjust  options  are  confusing. 


Task  3 

Nar  -  The  resp.  is  at  the  variance  trends  screen,  which  is 
where  the  last  task  ended.  The  resp.  chooses  the 
"utilities"  option  and  is  faced  with  the  utilities 
explanation  screen  and  menu  options. 

The  resp.  chooses  the  "interactive  update"  option, 
which  is  the  correct  choice.  The  resp.  then  faces 
the  interactive  update  explanation  screen  and  menu 
options.  The  resp.  then  chooses  the  "correct" 
option,  which  is  correct  for  the  task. 

Resp  -  This  is  something  like  Lotus  123. 

Nar  -  The  resp.  finishes  the  task. 


Task  4 

Nar  ~  The  resp.  is  at  the  spreadsheet  screen,  which  is 
where  the  last  task  ended.  The  resp.  chooses  the 
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Resp 

Rschr 

Resp 

Rschr 

Nar 


Resp 

Rschr 

Nar 


"add  element"  option  from  the  menu  and  fills  in  the 
data  the  program  asks  for. 

Did  I  misspell  that? 

No,  that  is  a  bug  in  the  program.  Go  ahead  and 
answer  yes . 

Did  my  1.2.1  get  put  in?  No  it  isn't  in  there. 

You  might  want  to  go  back  to  the  utilities  screen 
again.  We  didn't  really  get  out  of  the  other  task. 
The  resp.  chooses  the  "finished  option"  and  the 
utilities  menu  is  displayed.  The  resp.  chooses  the 
"interactive"  option  which  is  correct.  The  resp. 
then  chooses  the  "correct"  option  after  some 
confusion . 

I  guess  we  should  try  the  Mar84  time  frame  and  see 
what  happens . 

There  you  go,  it  has  your  new  element  in  there  now. 
The  resp.  then  fills  in  the  data  and  finishes  the 
task . 


Scale 


No  Oral  Comments 


Closing  Remarks 

Q  #1  -  Air  Force  familiarization  courses  in  AMS,  Wang,  and 
Zenith  familiarization. 

Q  #2  -  Word  processing  &  viewgraphs . 

Q  #3  -  Probably  not  relevant  to  job. 

Q  #4  -  10  minutes . 

Q  #5  -  The  program  would  be  more  relevant  to  program 
managers . 


Transcription  CAPPS  B 

Nar  -  The  resp.  starts  at  the  overview  screen. 

Resp  -  The  bracketed  one  should  be  the  next  chosen. 

Nar  -  The  resp.  chooses  the  correct  sequence  and  arrives 
at  the  status  screen  as  per  scenario  directions. 
Resp  -  I  like  the  highlighting. 


Task  IB 

Nar  -  The  resp.  is  at  the  status  report  screen,  which  is 
where  the  last  task  ended.  The  resp.  presses  the 
"cumulative  performance  option,  followed  by  the 
"variance  trends"  option  and  finishes  the  task. 


94 


Task  1C 


Nar  -  The  resp,  is  at  the  variance  trends  screen,  which  is 
where  the  last  task  ended. 

Resp  -  I  don't  even  see  EAC . 

Nar  -  The  resp.  chooses  the  "indices"  option  and  then  sees 
the  EAC  option.  He  then  chooses  that  option  and 
finishes  the  task. 


Task  2A 

Nar  -  The  resp.  is  at  the  point  where  the  last  task  ended. 
The  EAC  screen  is  displayed. 

Resp  -  If  I  use  "quit",  do  I  go  all  the  way  back  to  the 
beginning? 

Nar  -  The  resp.  chooses  the  "next  key  element"  option.  He 
then  tries  a  couple  of  other  options  before  arriving 
at  the  correct  point.  He  then  finishes  the  task. 


Task  2B 

Nar  -  The  resp.  is  at  the  point  where  the  last  task  ended. 
The  screen  shows  the  status  report  of  the  element  in 
question . 

Resp  -  I  believe  zoom  is  the  last  six  months. 

Nar  -  The  resp.  chooses  the  cumulative  performance, 

variance  trends,  and  zoom  options  to  complete  the 
task . 


Task  3 

Nar  -  The  resp.  is  at  the  point  where  the  last  task  ended. 
The  variance  trends  screen  is  displayed.  The  resp. 
chooses  the  "overview"  option  and  then  the  "select 
element"  option,  the  latter  being  incorrect.  The 
resp.  then  uses  the  "overview"  and  "utilities" 
option  to  arrive  at  the  explanation  screen  for  the 
utilities. 

Resp  -  It  says  to  correct,  so  there  is  correct...  I  think 
I'll  use  "correct." 

Nar  -  The  resp.  corrects  what  the  scenario  calls  for  and 
completes  the  task. 


Task  4 


Nar  -  The  resp.  is  at  the  point  where  the  last  task  left 
off.  The  spreadsheet  screen  is  displayed.  The  resp. 
uses  the  finished  option  and  that  takes  him  back  to 
the  utilities  explanation  screen. 
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Resp 


Nar 


Resp 

Rschr 

Nar 

Resp 


I  find  this  one  a  little  confusing.  I've  got  an 
option  to  remove  part  of  a  WBS . . .  I  assume  there 
should  be  one  to  add  a  part,  besides  just  the  next 
period  of  data. 

The  resp.  uses  the  "correct"  option  and  begins  to 
add  the  new  element.  The  bug  in  the  program  shows 
up. 

I  don't  think  I  typed  that. 

I  think  that  is  a  bug  in  the  program. 

The  resp.  proceeds  and  finishes  the  task. 

Well,  that  is  a  neat  program. 


Scale 


No  expanding  comments 


Closing  Remarks 


Q  #1 
Q  #2 
Q  #3 
Q  #4 
Q  #5 


Computer  Education  class  and  Fortran  a  long  time  ago. 
Doesn’t  use  a  computer  very  often.  I  don't  have  one. 
15-20  minutes 

Useful,  but  not  for  me  right  now. 

No  more  comments. 


Translation  CAPPS  C 


Task  lA 

Nar  -  The  overview  screen  is  displayed.  The  resp.  uses  the 
"all  element"  option  and  chooses  the  desired 
element . 

Resp  -  I  guess  I  would  print  this  to  answer  the  first 
question . 

Rschr  -  Right . 


Task  IB 

Nar  -  The  status  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  then  chooses  the 
"cumulative  performance"  option  and  ends  the  task 
by  choosing  the  "variance  trends"  option. 


Task  1C 

Nar  -  The  variance  trends  screen  is  displayed,  which  is 

where  the  last  task  ended.  The  resp.  goes  off  track 
awhile  by  choosing  the  "status"  and  "overview" 
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options.  The  resp.  backtracks  through  the 
"cumulative  performance",  "variance  trends",  and 
"Indices"  options  to  end  the  task. 


Task  2A 


Nar  -  The  EAC  screen  is  displayed 
task  ended.  The  resp,  finds 
Resp  -  This  is  the  chart  to  print. 
Rschr  -  OK. 

Nar  -  The  resp .  ends  the  task  by 
report . 


,  which  is  where  the 
the  way  to  element 


printing  the  status 


1 


last 

.6. 


Task  2B 

Nar  -  The  status  report  screen  is  displayed,  which  is 
where  the  last  task  ended.  The  resp.  chooses  the 
appropriate  options  and  finishes  the  task. 


Task  3 

Nar  -  The  variance  trends  screen  is  displayed,  which  is 

where  the  last  task  ended.  The  resp,  has  a  hard  time 
realizing  he  needs  to  view  the  overview  screen  to 
start  the  task.  The  resp.  tries  using  the  element 
options  before  going  to  the  overview  screen  and 
choosing  the  "utilities  option.  The  utilities 
explanation  screen  is  displayed  and  the  resp. 
chooses  the  "interactive"  option. 

Resp  -  I  want  to  correct  this. 

Nar  -  The  resp  chooses  the  "correct"  option.  The  resp. 
tries  to  move  to  the  area  on  the  spreadsheet. 

Resp  -  I  can't  move  so  the  highlight  box  shows  the  numbers 
I  want . 

Nar  -  The  resp.  then  uses  the  arrow  keys  and  accomplishes 
the  task. 


Task  4 


Nar 


Rschr 

Resp 

Nar 


The  spreadsheet  screen  is  displayed,  which  is  where 
the  last  task  ended.  The  resp.  chooses  the  "finish" 
option  and  goes  back  to  the  utilities  explanation 
screen.  The  resp.  reads  the  screen  in  search  for  the 
appropriate  option  and  chooses  the  "modify  option, 
which  is  not  appropriate.  The  resp.  then  chooses  the 
"correct"  option  and  proceeds  to  enter  the  data  as 
called  for. 

Is  your  element  in  there  yet? 

1.1.2. . .  No. 

The  resp.  then  tries  to  find  the  new  element  by 
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going  back.  The  resp.  finds  the  new  element,  but  in 
doing  so,  he  somehow  erased  the  numbers  in  the 
database.  This  was  not  realized  until  later  in  the 
interview.  The  resp.  finishes  the  task  by  adding  the 
data  that  is  called  for. 

Resp  -  I  was  using  the  hit  or  miss  method  and  I  should  have 
been  using  the  manual . 

Rschr  -  That  is  one  way  to  do  it,  but  the  hit  or  miss  is 
another . 


Scale 

Q  #7  -  The  ability  to  correct  errors  I  had  a  problem  with. 

Again...  I  don't  know  how  much  at  fault  this  is... 
it  may  be  my  inabilities. 


Concluding  Remarks 


Q  #1 
Q  #2 
Q  #3 
Q  #4 
Q  #5 


System  Command  introduction  to  Graftalk  and  Wordstar. 
I'll  be  using  it  some. 

I  think  it  is  highly  relevant  for  program  management. 
About  15  minutes. 

I  think  the  first  time  user  will  be  confused  somewhat 
in  how  to  get  around.  I  have  a  problem  with  the 
program.  It  is  too  easy  to  destroy  the  database.  If 
you  make  a  change,  it  should  keep  the  old  and  ask  if 
it  should  delete  it. 


Transcript i on  CAPPS  D 
Task  lA 

Nar  -  The  overview  screen  is  displayed  for  the  start . 

The  resp.  chooses  the  "all  elements"  option  and 
finds  the  one  the  scenario  calls  for.  The  resp. 
then  finishes  the  task. 


Task  IB 


Nar  -  The  status  report  screen  is  displayed,  which  is 
where  the  last  task  ended. 

Resp  -  Whoops...  I  don't  want  that. 

Rschr  -  You  just  hit  the  arrow  key? 

Resp  -  Yes. 

Rschr  -  What  do  think  about  the  way  they  move  around. 

I  can  see  your  probably  used  to  using  a 
spreadsheet . 
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Resp 


Nar 


Rschr 

Nar 


Yah,  your  right.  I'm  just  used  to  using  arrow 
keys ,  The  spacebar  is  alright ,  it  is  just  not 
what  I'm  used  to. 

The  resp.  had  lost  the  element  that  was  needed, 
so  he  went  back  to  the  desired  element.  The  resp. 
uses  the  arrow  keys  again  to  try  and  move  in  the 
menu. 

I  can  see  your  used  to  arrow  keys. 

The  resp.  chooses  the  "cumulative  performance" 
option  and  then  the  "variance  trends"  option  to 
finish  the  task. 


Task  1C 

Nar  -  The  variance  trends  screen  is  displayed,  which  is 
where  the  last  task  ended.  The  resp.  gets  off 
track  by  going  back  to  the  status  screen  for  the 
whole  program. 

Rschr  -  Your  in  element  one  which  is  the  whole  program. 

You  need  to  get  back  to  the  element  you  were  in. 

Nar  -  The  resp.  then  finds  the  way  back  to  the  desired 
element.  The  resp.  then  chooses  the  correct  path 
to  the  EAC  screen  to  finish  the  task. 


Task  2A 


Nar 

Resp 

Rschr 

Resp 

Nar 


The  EAC  screen  is  displayed,  which  is  where  the 
last  task  ended.  The  resp.  chooses  the  "quit" 
option  in  an  effort  to  get  to  a  screen  with 
options  for  a  change  in  element  selection. 

Thanks  for  using  CAPPS  ...  laughs . 

Did  you  think  it  was  going  to  quit  that  one 
option  and  put  you  back  on  the  main  screen 
again? 

Well...  I  was  lost  and  didn't  see  what  I  wanted 
or  I  didn't  recognize  how  to  get  back  to  your 
option  screen.  I  would  have  preferred  some 
instructions  there. 

The  resp.  then  loaded  the  program  again  and  found 
the  desired  element  and  finished  the  task. 


Task  23 

Nar  -  The  status  report  screen,  which  is  where  the  last 
task  ended  is  displayed.  The  resp.  then  works  to 
the  variance  trends  through  the  "cumulative 
performance"  option. 

Rschr  -  Does  that  confuse  you  or  does  the  menu  showing 

the  "variance  trends"  option  grab  your  attention 
more  than  the  title? 
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Resp  -  Well...  I  was  focusing  down  there.  They  could 
throw  a  color  combination  up  there  that  could 
reinforce  where  your  at, 

Nar  -  The  resp.  then  chooses  the  "zoom"  option  to 
finish  the  task. 


Task  3 

Nar  ~  The  variance  trends  screen  is  displayed,  which  is 
where  the  last  task  ended.  The  resp.  then  chooses 
the  "utilities"  option  and  views  the  explanation 
screen.  The  resp.  chooses  the  "interactive"  and 
"correct"  options  and  finishes  the  task. 


Task  4 

Nar  ^  The  spreadsheet  screen,  which  is  where  the  last 
task  ended,  is  displayed.  The  resp.  chooses  the 
"add  element"  option  and  adds  the  information 
that  is  asked  for.  The  bug  in  the  program  is 
explained  and  the  resp.  continues. 

Rschr  -  It  looks  like  you  misspelled  it  doesn't  it. 

Resp  -  Yes . 

Rschr  -  You  didn't.  I  think  it  is  a  bug  in  the  system. 

Did  it  put  the  new  element  in  for  you? 

Resp  -  No,  it's  not  there. 

Nar  -  The  resp.  goes  back  to  the  explanation  screen  and 
then  the  new  element  appears. 

Rschr  -  What  do  think  about  that?  That  could  be 

confusing.  What  do  you  think  should  happen? 

Resp  -  If  you  can’t  follow  this  completely  through  on 
the  one  screen  when  you  build  on  your  database, 
it  should  tell  you  on  that  or  the  next  screen. 


Scale 

Q  #6  -  I  had  no  problem  with  it. 

Q  #9  -  It's  probably  alright  compared  to  others. 


Concluding  Remarks 


Q  #1  - 


Q  #2  - 
Q  #3  - 

Q  #4  - 
Q  #5  - 


Air  Force  introduction  to  microcomputer 
applications.  I  had  a  college  course  in  computer 
theory  and  simple  programming. 

I  use  Lotus  alot  almost  exclusively. 

It  is  not  that  relevant  to  me,  but  I  can  see  it 
for  others. 

1  hour. 

No  comment . 
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Transcription  CAPPS  E 
Task  lA 

Nar  -  The  resp.  is  viewing  the  overview  screen.  The 
.  resp.  then  chooses  the  "all  element"  option. 

Resp  -  I  don't  care  for  these  (spacebar  and  backspace 
keys),  I'm  used  to  using  the  arrow  keys  to  go 
back  and  forth . 

Nar  -  The  resp.  finds  the  appropriate  element  and 

displays  the  status  report,  thereby  finishing  the 
task . 

Resp  -  That  wasn't  too  bad  to  find  in  here. 


Task  IB 


Nar 


Resp 


Rschr 

Resp 

Nar 


Resp 

Rschr 

Resp 

Rschr 

Resp 

Nar 

Resp 

Rschr 

Resp 


Rschr 

Resp 


The  resp.  is  at  the  point  where  the  last  task 
ended.  The  status  report  screen  is  displayed.  The 
resp.  tries  the  "explain"  option.  The  resp.  then 
tries  to  use  the  "previous  screen"  option. 

Can't  go  back  any  further.  That's  bad,  I  had  to 
press  this  three  times  Just  to  be  able  to  read 
it.  I  guessed  at  it. 

Too  quick  a  system  message? 

Yes...  Way  too  quick.  It  should  stay  on  there 
until  you  hit  whatever. 

The  resp.  then  tries  the  "cumulative  performance" 
option,  which  is  the  correct  path.  The  menu  shows 
the  option  "variance  trends"  among  others. 

It's  that...  right? 

No,  your  on  cumulative  performance. 

Oh,  OK  variance  trends...  OK. 

Do  you  have  a  hard  time  seeing  the  title  on  the 
top  verses  the  menu? 

I  think  I  tend  to  focus  more  on  this  (the  menu) . 
The  resp.  chooses  the  "variance  trends"  option, 
and  finishes  the  task. 

Why  don't  they  Just  put  print  or  output  in  the 
menu? 

There  is  no  print  programming  in  the  program.  It 
Just  uses  what  is  available  with  the  computer. 
Most  systems  that  I've  used  will  take  you  to  a 
file  system  with  options  and  then  it  prints  it 
right  out . 

Do  you  think  there  should  be  a  print  function? 

I  would  put  one  in. 


Task  1C 

Nar  -  The  resp.  is  at  the  point  where  the  last  task 

ended.  The  variance  trends  screen  is  displayed. 
The  resp.  then  decides  to  proceed  with  the 
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“indices"  option.  This  shows  the  desired  "EAC" 
option  and  the  resp.  finishes  the  task. 


Task  2A 

Nar  -  The  screen  displayed  is  the  EAC  screen,  which  is 
where  the  last  task  ended.  The  resp.  chooses  the 
“overview"  option  and  the  “all  elements"  option 
to  reach  the  status  report  of  the  desired  element 
and  finish  the  task. 


Task  2B 

Nar  -  The  screen  displayed  is  the  status  report  screen, 
which  is  where  the  last  task  ended.  The  resp. 
chooses  “cumulative  performance",  “variance 
trends",  and  “zoom"  options  and  finishes  the 
task . 


Task  3 


Nar  -  The  resp.  chooses  the  “overview"  option  and 

leaves  the  variance  trends  screen  that  ended  the 
last  task.  The  resp.  chooses  the  “utilities" 
option  and  the  "interactive  update"  option  to 
enter  the  database.  The  resp.  chooses  the 
"correct"  option  and  enters  the  spreadsheet  and 
corrects  the  mistake  as  per  scenario  directions. 


Task  4 


Nar 


Resp 

Rschr 

Nar 

Resp 

Nar 


Resp 


The  screen  displayed  is  the  spreadsheet  screen, 
which  is  where  the  last  task  ended.  The  resp. 
moves  to  the  database  option  explanation  screen. 
The  resp.  then  chooses  the  "correct"  option  and 
the  spreadsheet  is  displayed. 

This  is  where  1  want  to  be  to  add  a  new  element 
to  the  LRP? 

Go  ahead  and  try  out  what  you've  got  there  and 
see  if  you  can  add  a  new  element. 

The  resp.  presses  the  slash  key  to  display  the 
menu . 

There  is  "add  element". 

The  resp.  chooses  the  "add  element"  option  and 
adds  the  required  information  the  program  calls 
for.  The  resp.  passes  over  the  bug  in  the  program 
without  noticing  it  and  finishes  the  task  by 
entering  the  correct  data. 

Spreadsheets  are  the  best  thing  going.  1  use  them 
all  the  time.  Is  this...  I  need  to  ask.  Blue  and 
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yellow  is  what  I  always  use.  Can  you  change  the 
colors  in  CAPPS. 

Rschr  -  I  think  this  is  about  it  for  this  program. 

Resp  -  The  color  isn't  controlled  by  the  operating 
system . 

Rschr  -  It  is  controlled  by  the  software. 

Resp  -  I'd  rather  have  blue  and  yellow.  White  drives  my 
crazy.  The  blue  is  alright  but  white  bothers  my 
eyes  after  awhile. 


Scale 


No  Expanding  comments. 


Concluding  Remarks 


Q  #1 

Q  #2 
Q  #3 

Q  #4 


Three  courses  in  graduate  school .  They  were  in 
basic  programming  and  information  systems. 

Yes,  at  least  50%  of  the  time. 

It  is  pretty  good.  I  work  with  someone  that  uses 
it  all  the  time.  It  is  relevant. 

About  5  minutes 


Transcription  CAPPS  F 


Task  lA 


Nar 


Resp 


Resp 

Rschr 

Resp 

Rschr 

Resp 


Nar 

Reap 

Rschr 

Resp 


The  resp.  is  looking  at  the  overview  screen  before 
starting  the  task.  The  following  is  in  regard  to 
the  menu. 

One  thing,  I  think  what  works...  I  guess  it  takes 
just  getting  used  to  it,  but  I  like  when  it 
highlites  with  a  different  color.  You  can  see 
this,  but  to  me  it  would  be  alot  quicker  and  more 
contrasting  if  just  a  different  color. 

Do  these  arrow  keys  work. 

Go  ahead  and  try  it . 

No.  So  you  just  have  to  page  through  them  if  you 
want  to  get  back  to  where  you  were. 

Backspace  works  also.  Would  you  rather  use  the 
arrow  keys? 

Here  again,  it  is  what  I'm  used  to.  You  use  arrow 
keys  in  going  back  and  forth  in  Enable.  It  isn't 
right  or  wrong,  it's  just  different. 

The  resp.  is  looking  at  the  status  report  screen. 
This  kind  of  stuff,  I  like  that.  That  really  jumps 
off... 

You  like  the  highlighting  of  the  text  there. 

Yes.  How  would  you  print  it? 
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Rschr  -  You  would  use  a  Control  G  or  something  like  that, 
or  shift  printscreen. 

Resp  -  Why  couldn't  they  put  a  print  in  the  menu?  Here 

again  in  some  of  the  other  common  programs,  one  of 
the  options  is  always  print . 


Task  IB 


Nar 

Resp 

Rschr 

Nar 

Resp 

Rschr 

Resp 


The  resp.  is  at  the  status  report  of  the  element 
in  question  from  task  lA. 

How  do  you  go  back  to  a  previous  screen? 

You  can  in  some,  but  not  in  this  case.  You  would 
have  to  use  the  next  key  element  option. 

The  resp.  pushes  the  cumulative  performance  option 
and  that  graph  and  menu  is  displayed. 

I  really  think  like  here  (points  to  menu) ,  if  this 
was  highlighted  in  a  different  color,  it  wouldn't 
run  in  with  status.  I  would  then  print  it. 

No,  your  on  the  cumulative  performance  chart.  You 
want  variance  trends.  Do  you  find  that  menu 
grabs  your  attention  more  than  that  title  does? 

Ah  no,  it's  just  stupidity  that  got  me  that  time. 


Task  1C 

Nar  -  The  resp.  is  still  viewing  the  variance  trends 
chart  from  the  previous  task. 

Resp  -  Zoom  and  expand  are  physical  things  aren't  they? 

They  have  nothing  to  do  with  estimating.  Indices, 
I  don't  understand  what  they  are. 

Nar  -  The  resp.  presses  the  indices  option  and  the 

option  with  the  desired  result  is  displayed.  The 
resp.  then  presses  the  zoom  option. 


Task  2A 

Nar  -  The  resp,  is  at  the  previous  task  screen  which 
shows  the  zoom  screen  from  task  1C.  The  resp. 
chooses  the  key  element  option  to  find  the 
element  needed  by  the  scenario.  The  resp.  then 
accomplishes  the  task  by  printing  the  status 
report . 


Task  2B 

Nar  -  The  resp.  is  at  the  point  in  the  program  where  the 
previous  task  left  off.  The  status  report  is 
displayed.  The  resp.  proceeds  to  the  variance 
trends  screen  where  the  menu  is  displayed. 
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Rschr 

Resp 


Rschr 


Resp 


Nar 


Resp 

Nar 

Resp 

Nar 

Rschr 

Resp 

Rschr 

Resp 


Now  how  do  you  show  only  the  last  six  months? 
Adjust...  or  expand...  or  zoom...  I  don't  know 
what  that  one  does...  status.  See,  none  of  those 
makes  sense.  You  would  want  a  reduce,  not  an 
expand . 

-If  you  want  to  focus  on  the  last  six  months 
instead  of  twelve,  which  one  of  those  options 
would  work? 

Well,  it  has  to  be  that  one  I  guess  (the  status 
option) .  It  seems  to  me  none  of  them  makes  sense. 
Can  I  come  back  to  this  easy? 

The  resp.  tries  the  status  option  and  goes  off 
track  for  a  few  minutes  until  returning  to  the 
variance  trends  screen.  The  resp.  then  attempts  to 
use  the  documentation. 

Oh  lets  just  try  it. 

The  resp.  chooses  the  indices  option,  and  again 
has  to  work  back  to  the  variance  trends  screen  and 
associated  menu. 

That  is  one  of  my  comments  then.  To  me,  reading  it 
like  this  without  knowing  it...  it  seems  like 
there  should  be...  it  isn't  intuitively  obvious 
which  you  use  to  display  the  last  six  months. 

The  resp.  tries  the  adjust  option  and  nothing 
changed.  The  resp.  then  tries  the  zoom  option  and 
completes  the  task. 

You  don't  think  zoom  is  intuitively  obvious? 

No...  Well,  I  guess  now  that  I  know  that  is  the 
one,  I  guess  that  is  logical. 

Maybe  they  should  have  an  option  titled  the  "last 
six  months . " 

No,  then  it  would  be  cluttered.  I  would  say,  they 
should  spread  this  out  a  little  more  (Resp.  refers 
to  the  menu  on  the  variance  trends  screen) ,  wether 
you  use  color  or  not.  They  are  just  wasting  space 
out  here,  why  not  use  it? 


Task  3 


Nar 


Rschr 

Resp 


Nar 


The  resp.  starts  from  the  variance  trends  screen 
from  the  previous  task.  The  resp.  chooses  the 
overview  option  and  the  overview  screen  and  menu 
is  displayed. 

Which  of  these  options  will  allow  you  to  change 
the  database? 

I  don't  know  if  you  want  the  utilities...  what  1 
think  you  should  do  is  to  go  to  the  option  that 
allows  you  to  select  the  element  you  want.  Then 
you'll  have  a  menu  that  lets  you  add  or  modify  it. 
The  resp.  chooses  the  "select  and  element"  option. 
The  screen  shows  all  of  the  elements. 
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Resp  - 


Rschr 

Resp 

Nar 


Rschr 

Resp 


Rschr 

Resp 

Nar 


Resp  - 
Rschr  - 

Resp  - 


I  guess  you  can’t  go  highlight  the  one...  you  have 
to  put  it  in  there.  The  resp.  types  the  number  in 
and  the  status  screen  is  displayed. 

How  are  you  going  to  change  anything  from  there? 
Hmm...  it  doesn't  look  like  I  will. 

The  resp.  tries  the  "all  element"  option  and  then 
in  a  few  minutes  goes  to  the  overview  screen  and 
chooses  the  "utilities"  option,  which  is  the 
correct  path.  The  screen  displays  the  text 
explanation  of  the  utility  functions  as  well  as 
the  menu  for  those  options.  The  resp.  tries  the 
modify  option  and  not  seeing  a  way  from  there, 
chooses  the  "correct"  option  which  is  appropriate. 
Do  you  think  they  ought  to  get  rid  of  "modify"? 

To  me,  that  is  the  one  you  would  want  to  press  if 
you  wanted  to  change  something.  It  is  not 
necessarily  a  mistake. 

It  is  too  close  to  the  same  meaning  as  correct? 
Yes,  it  is  not  clear  which  does  what. 

The  resp.  uses  the  arrow  keys  to  locate  the 
mistake  and  corrects  the  amount  as  the  scenario 
requested.  The  resp.  then  tries  to  leave  the 
"correct"  mode  by  pressing  the  key  that  displays 
the  menu.  The  resp.  then  tries  to  use  the  arrow 
key  . 

It  doesn't  work  in  that  one. 

Do  you  thing  that  the  same  method  should  be  used 
to  move  around? 

Yah,  I  would  think  consistency  is  something  good. 


Task  4 


Resp  - 

Rschr  - 
Resp  - 
Nar  - 


Resp  - 


The  new  task  is  begun  at  the  same  screen  as  the 
last  task  left  off  from.  The  menu  on  the 
spreadsheet  is  displayed.  The  resp.  chooses  the 
"add  new  element"  option  and  enters  the 
information  the  program  calls  for.  The  program 
then  displays  the  name  of  the  element  as  it  always 
does  at  this  point.  There  is  no  space  between  the 
name  and  the  resp.  wonders  if  there  was  a 
misspelling.  The  resp.  is  told  about  the  bug  and 
proceeds  by  answering  yes  to  the  posed  question. 

So  now  we  just  need  to  put  in  the  rest  of  the 
figures . 

Lets  see  if  the  new  element  name  is  displayed. 

No  it's  not.  What  happened? 

The  resp.  brings  up  the  menu  and  chooses  the 
"finished"  option.  The  spreadsheet  then 
initializes  and  displays  the  element  just  entered. 
It  asked  if  I  was  done  and  I  answered  yes.  It 
should  tell  you  to  go  back  to  this  screen 
(utility  explanation)  if  you  need  to. 


Nar 


Scale 


Q  #1 


Q  #2 
Q  #3 
Q  #4 

Q  #5 


.  Q  #6 
Q  #7 


Q  «8 
Q  #9 


Q#10 

Q#ll 


Closi 

Q  #1 
Q  #2 

Q  #3 
Q  #4 
Q  #5 


The  resp.  then  enters  the  appropriate  numbers  and 
finishes  the  task. 


The  contrast  with  the  background  is  good  overall 
except  with  the  menu.  They  should  highlight  that 
with  color. 

The  same  comment  applies  to  overall  use  of  color. 
I  didn't  use  it  so  I  have  no  comment. 

The  feedback  from  the  messages...  what  I  saw  was 
fine . 

The  overall  screen  layout  was  fine  except  for  a 
couple  of  menus  that  I  thought  were  too  close 
together  and  that  was  a  little  bit  confusing  for 
the  uninitiated,  but  other  than  that  is  was  fine. 
The  graphics  were  fine. 

Compared  with  enable,  this  is  more  cumbersome.  No 
matter  where  you  are,  you  can  go  back  to  the  last 
screen.  Compared  with  that,  this  is  average. 

It  is  not  terrible,  but  it  is  not  good  either. 

The  written  documentation  looked  alright  for  what 
I  looked  at  it.  1  would  prefer  a  more  detailed 
index  in  the  back. 

Pretty  good. 

That  is  fine  I  guess. 


Comments 


No,  not  even  from  the  Air  Force. 

Some  word  processing,  viewgraph  and  database 
work . 

About  20  minutes. 

Maybe . 

I  don't  know  if  this  would  be  useful,  but  people 
tend  not  to  read  things.  They  don't  have  time, 
especially  in  our  place.  Some  applications  have  10 
minute  guides  to  tell  how  to  use  something.  It  is 
like  a  three  page  pamphlet  or  something.  It's  like 
anything  else,  people  are  not  going  to  read  the 
detailed  instructions,  especially  in  a  case  like 
this  where  it  is  menu  driven  to  a  large  extent. 
They  will  just  experiment  because  it  is  easier  to 
do  it  that  way  and  they  will  learn  it  by  going 
ahead  and  doing  it.  If  you  had  a  real  simple  guide 
with  just  the  basics,  it  might  help. 
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Task  lA 


NAR 

Resp 

Rschr 

Resp 

Rschr 

Resp 


Rschr 

Resp 


The  resp.  has  just  chosen  the  key  element  option 
.from  the  overview  screen. 

We  can  see  which  element  has  the  largest  cost 
variance . 

We  need  to  print  a  status  report  of  that. 

Hmm  spacebar  .... 

You  just  went  to  a  different  element. 

Oh  ok,  I  understand,  I  see  where  we  are  at.  The 
white  shows  us  what  element  we  are  looking  at.  We 
would  want  to  print  this. 

No,  your  on  the  right  element,  but  you  want  to  find 
the  status  report . 

Oh,  that  is  our  status  report.  Print  it. 


Task  IB 


NAR 

Rschr 

Resp 

NAR 

Resp 

Nar 

Resp 

Rschr 

Resp 

Rschr 

Resp 


The  resp.  is  at  the  point  in  the  program  where  the 
status  report  of  the  task  1  element  is  on  the 
screen . 

The  next  thing  you  want  to  do  is  print  a  graphical 
chart  showing  the  variance  trends.  That  is  your  menu 
on  the  bottom.  You  just  moved  across  the  screen  with 
the  space  bar. 

The  next  key  element,  lets  see  what  that  does  for 
us  . 

Resp.  realized  that  option  would  not  work. 

Cumulative  performance,  lets  see  what  that  does  for 
us  . 

The  resp  looks  at  the  cumulative  performance 
screen  menu  and  associated  screen. 

This  is  what  we  are  asked  for,  the  variance 
trends . 

Your  on  the  cumulative  performance  chart.  You  want 
to  do  the  variance  trends. 

How  do  you  know  which  one  is  shown? 

Here  is  a  question  for  you.  Do  you  find  the 
bracketed  menu  item  strikes  your  eye  more  than  the 
chart  title? 

No,  the  brackets  don't  jump  out  at  me.  I  assume 
now. . .  1  notice  the  brackets  now,  but  before  I 
didn't.  I  guess  now  this  tells  me  I  would  be 
selecting  the  variance  trends,  but  since  it  is  the 
same  color  as  everything  else,  it  doesn't  really  jump 
out  at  me.  In  fact  if  you  didn't  notice,  I  was  kind 
of  fooling  around  because  I  didn't  know  where  I  was 
at.  Here  I  can  see  that  this  is  the  cumulative 
performance  and  this  tells  me  what  has  been  printed 
out  on  the  screen.  Now  I  was  looking  for  variance 
trends  and  I  didn't  even  notice  it  down  there.  I 
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think  it  would  be  helpful  if  it  were  color  coded, 
just  like  the  graph,  where  if  things  are  not 
selected,  they  are  one  color  and  the  option  about  to 
be  selected  is  another  color. 

Nar  -  Hesp .  presses  variance  trends  option  and  the  screen 
is  displayed. 

Resp  -  Now  this  is  the  part  we  are  looking  for. 


Task  1C 

Nar  -  The  variance  trends  screen  is  still  displayed  for 
the  element  we  are  interested  in. 

Resp  -  We  are  looking  for  estimates.  Lets  see  what  we  got 
to  play  with  down  here. 

Nar  -  The  resp.  presses  the  overview  option  and  then  the 
status  and  cumulative  performance  options  for  the 
whole  LRP  program. 

Rschr  -  Your  now  looking  at  the  whole  program  and  your 
little  element  is  gone. 

Resp  -  Oh,  we  got  out  of  it  somehow.  Your  right,  we'll  go 
back  there . 

Nar  -  Resp  finds  the  EAC  screen. 

Resp  -  This  is  friendly.  Obviously  I  haven't  read  the 
manual  in  depth...  it  is  user  friendly. 


Task  2A 

Nar  -  The  resp.  proceeded  to  the  key  variance  screen  and 
chose  the  a-^propriate  element. 

Resp  -  We  need  the  status  report  and  here  is  the  status 
report . 


Task  2B 

Nar  -  The  resp.  has  just  finished  task  2A  and  the  screen 
shows  the  status  report.  The  resp.  goes  one  screen 
past  the  one  that  is  desired. 

Resp  -  Can  I  go  back  to  the  last  page? 

Rschr  -  Once  you  pass  it  you  can't  go  directly  back. 

Resp  -  I  think  that  is.... 

Rschr  -  Would  you  like  the  option  of  going  back  to  the 
previous  page? 

Resp  -  Oh  yah,  sure. 

Nar  -  The  resp  works  around  from  the  beginning  of  the 
process  to  the  screen  that  is  desired. 

Rschr  -  Now  your  at  variance  trends  again,  how  can  you  make 
it  show  just  the  last  six  months? 

Resp  -  Maybe  we  could  use  zoom. 

Nar  -  The  reap,  uses  the  zoom  option  and  completes  the 
task . 
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Task  3 


Nar 


Rschr 

Resp 

Rschr 

Resp 

Nar 

Rschr 

Resp 

Rschr 

Resp 

Rschr 

Resp 


Nar 

Resp 

Nar 

Rschr 

Resp 

Nar 


The  resp,  went  looking  for  the  option  to  change 
database  entries  in  the  element  report  options 
before  going  to  the  database  utilities  option.  The 
-screen  that  explains  the  database  functions  is 
displayed . 

You  want  to  correct  an  error,  so  which  one  of  those 
things  should  you  choose? 

Modify  or ... . 

Are  the  terms  modify  and  correct  confusing? 

No.  Not  after  you  read  it  all.  I  can  see  where  it 
might  be  confusing  to  some  people. 

The  resp.  uses  the  "correct"  option  listed  on  the 
menu.  The  spreadsheet  is  displayed. 

What  did  you  do  to  make  that  beep? 

I  hit  the  space  bar  to  try  and  move  over. 

You  just  hit  the  space  bar,  what  would  you  normally 
hit  using  a  spreadsheet? 

The  cursor  or  arrow  keys. 

You  went  from  one  way  to  another.  Would  you  prefer 
they  were  all  the  same? 

Consistency,  yah  when  you  use  a  spreadsheet,  the 
arrow  keys  always  move  you  around.  We  have  been 
using  the  space  bar.  My  first  impulse  was  to  try  and 
move  over  so  I  went  to  the  space  bar...  That  is  when 
I  got  the  error.  I  think  we  should  be  consistent  as 
to  how  we  move  around. 

The  resp.  hit  a  key  and  the  program  beeps. 

Unknown  function,  press  return  to  continue.  I  wonder 
if  1  could . . . 

The  resp.  corrects  the  cell  as  one  normally  does  on 
a  spreadsheet . 

What  did  you  do  to  make  the  noise? 

1  hit  this  key. 

The  resp.  was  trying  to  display  the  menu  like  the 
screen  suggested,  but  hit  the  back  slash  key  instead 
of  the  slash  key . 


Task  4 

Nar  -  The  resp.  is  at  the  previous  task  screen  and  chooses 
the  "finished"  option.  The  database  function  screen 
is  displayed.  The  resp.  reads  them  all  and  can't 
decide.  Finally  the  resp.  chooses  the  "correct" 
option . 

Rschr  -  Would  you  prefer  they  had  this  better  explained  on 
the  screen? 

Resp  -  Yah,  Z  think  you  lose  track.  1  had. 

Rschr  -  How  would  you  add  a  new  element? 
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Resp  - 
Nar  - 


Rschr  - 
Resp  - 
Rschr  - 


Nar  - 


Scale 


Q  #1  - 

Q  #3  - 
Q  #8  - 


Closing 

Q  #1  - 

Q  #2  - 
Q  #3  - 
Q  #4  - 
Q  #5  - 


Use  this  key  to  display  menu  is  the  first  thing  I 
would  do. 

The  resp.  presses  the  key  that  shows  the  menu. 

(The  backslash  key  was  pressed  first  and  then  the 
correct  "slash"  key).  The  resp.  then  presses  the  add 
new  element  option  and  adds  the  information  from  the 
scenario  when  asked  by  the  screen  prompts.  The 
system  then  displays  an  are  you  sure  message  with 
the  name  of  the  new  element .  The  element  name 
appears  to  be  misspelled. 

Does  it  look  like  you  misspelled  integration? 

Yes  . 

You  didn't.  It  is  a  bug  in  the  system  and  it  does 
that  to  everybody.  It  displays  the  system  name  right 
next  to  the  message  without  any  spaces. 

The  resp.  presses  the  "yes"  option  and  fills  in  the 
rest  of  the  required  data. 


Option  selections  at  the  bottom  of  the  screen  should 
be  color  coded. 

Some  definitions  need  to  be  more  clear  cut. 

Movement  controls  are  not  consistent.  Sometimes  the 
spacebar  is  used  and  then  at  other  times  the  arrow 
keys  are  used. 

Remarks 

Oh  ya,  computer  science,  microprocessors,  and 
Pascal,  Ada,  C,  assembly,  and  ba^ic  programming. 

Yes,  every  day.  Wordprocessing  and  viewgraphs. 

Not  at  all  . 

Very  relevant. 

I  think  the  program  has  great  possibilities.  I 
think  overall  the  system  is  friendly,  very 
friendly,  it's  flexible.  I  think  it  needs  to  be 
more  uniform,  more  consistent  in  how  you  move.  I 
think  some  of  the  options  need  to  be  better 
defined  and  clearer  cut,  such  as  the  modify  and 
correct  options.  I  think  they  may  confuse  users 
until  they  get  used  to  it  or  it  is  going  to  be 
geared  towards  a  specific  group  that  only  work  in 
that  environment . 


Ill 
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